CES 2010: NUI with Bill Buxton

A few quick comments on your post:

Yes, "painter's shoulders" is an issue.  If we are working on a kiosk or a whiteboard, things may be fine.   In the former, the interaction will likely be brief, and in the latter, we can get close enough that the arms need not be extended out so far as to cause excess fatigue.  On the other hand, when sitting, and reaching over a keyboard, desk, etc. to a vertical monitor, we can generally expect less use and more fatigue, due to extent of reach.  There is a reason that drafting tables were in a sloping desk rather than vertical wall format!

As for human skill, the interesting thing is that the essence of skill acquisition (and hence skill utilization and transfer) is the notion of "chunking".  (See  the following if you want more background:  http://billbuxton.com/chunking.html).  This prompts me to go beyond wondering if/when I should use touch or stylus, and rather - recognizing that I have two hands, and can use them together - consider when it is best to use them together:  for example, touch in my non-dominant hand, and stylus in my dominant one.  This reflects what we do in the everyday physical world, and provides a mechanism to "chunk" together low level tasks into a higher level one, using an existing skill.  We have to stop looking at technologies as being in competition.

As for operating devices while driving, I don't think that we have a disagreement - however, your comment suggests to me that I should make the point more strongly than I did.  You will note that in the interview I did say minimizing cognitive interferance as well as visual and manual.  As well, I qualified things by saying something, "to the extent that it can be done safely."  But you are right, while having the perfect voice only phone (let's pretend that there could be such a thing) sitting on the passenger seat, and speaking to a remote person using it, it is definately NOT the same as having the same conversation with a live person sitting in the same seat under the same conditions.  The real person has cose to the same ability to perceive the world and context in which you are driving as you, so knows why you go silent, or when they need to shut up for a moment.  Hence, the driver's ability to flow in and out of the conversation with minimal distraction/overhead, is much better than it would be with the person on the phone, who as no idea that something was about to happen that really needed your full attention.  The reality is, that there are a large number of things that already distract us from our primary task of driving (tuning the radio, adjusting the heat, sipping on a soda, listening to an interview on the radio, thinking about your spouse ...).  So the real question is, where do we draw the line?  The related question is this:  with an understanding of the issues above, by worthy design, how low can we get the interference of a remote conversation?  Can we get it below the threshold of acceptable risk?  For example, would augmenting the audio with video that gives the remote person a better sence of context help?  I'm not advocating a design, or even suggesting that there is an acceptable solution.  The fact is, unless I ask such questions, I can't answer the question.  But you are absolutely right:  to even begin, we have to be aware of the issue, or we won't begin asking them.  What I do know is that it is certainly unacceptable to text, read email, search for an address, etc. by any combination of voice and touch while driving.  We both know that.  Thanks for the reminder.

Thanks for the comments.

Bill

Josh

Just one short comment. 

I have no issues with your comments about my book - except for one:  in what way did I "trash Adobe"?  Why would I?  They are a great company - one very much worthy of respect.  That is why I used them as an example.  That is the same reason that I used Apple.   The message in the Adobe example was simple:  even outstanding and successful software companies have a poor record of developing new products in-house, and rely upon mergers and acquisitions for their growth.   Another reason that I used Adobe was that their data was available, and in a form  that readers could easilly check.  Finally, the argument that the example was intended to support was that if companies invested in a design process, then in-house development of new (as opposed to N+1) products would become another viable option for growth.  If there was criticism implied in the example, it applied to the software industry as a whole, not to any one company.  I thought that I made that clear.  If not, I apologize.

Good question on chunking w.r.t. cut/copy & paste.  It helps tease out some interesting issues.  The interesting thing about human skill is that we can - through practice - transform from attentive problem solving behaviour to automatic skilled behaviour.  Once acquired, we can do remarkable things.  The question then reduces to, "At what cost?"  So let's look at the basic action, MOVE as an example.  As we all know, there are at least 2 ways to do this with a GUI.   (1) cut-and-paste, and drag-and-drop.   From the perspective of skill acquisition, a key challenge with cut-and-paste lies in the concept of the clip-board. It s this magic place which is (for most) invisible.  In contrast, with drag-and-drop, the whole transaction is visible and there is a real-world metaphor.  I think that it is pretty clear that the former is harder to learn and more prone to error for beginners.  However, with practice, cut-and-paste and the clipboard will become a highly learned skill, and therefore be automatic, not require concious attention during execution.  That is, when a highly learned skill, it will impose no higher demand on the end user than drag-and-drop.  While both enable one to effect a MOVE, the two techniques have important differences.  The prevue capability of drag-and-drop makes it especially approrpriate for some graphical tasks, for example.  But, there is also confusion whether drag-and-drop means move (yes, if on same disk) or copy (if from one disk to another) - a conceptual "bump" that will hit users, frequently after they think that they "get it".  Cut-and-paste is less visible, but having grokked the concept of the clipboard, one can exploit the skill to effectively "move" something from one location, to multiple locations.  That is, the added complexity of the clipboard means that the move operation can be extended to multiple pastes. 

A quick summary up to here:  (1) even otherwise disconnected actions can chunk together to form a skill; (2) the UI design needs to consider impact on, and find a balance betwen user's acquisition of skill as well as performance once skill is acquired; (3) different ways to do the same thing, such as move, differ in the range of their semantics, and not just in how they are articulated.

A key point of my chunking & phrasing paper is to say that through what I called "kinesthetic connectivity" we can accelerate the aggregation of sub-tasks into the desired skill.  However, that does not neccessarily rule out being able to still take advantage of the individual atomic skills that chunk to constitute the higher order aggregate skill.  Paste as a component of the higher level skill Move in the cut-and-paste-with-clipboard method, is an example of this.

So far so good.  But up to now, I may seem to be avoiding your question, and stating the obvious.  So let's behave like designers and see how this approach to thinking about things might shed light on other ways to do things.  Can we augment current techniques, and/or find new and better techniques?

In our work in my old Input Research Group at the University of Toronto, we (students Garry Hardock and Gord Kurtenbach and myself) looked at this very example.  So the one thing that leaps out from my description above, is the initial problem of the invisibility of the clipboard.  Not only are its contents visibile, rarely is there any effective explicit visible representation of the clipboard itself - something rare in the GUI.  On the other hand, there is the problem disadvantage of the drag-and-drop metaphor, in that it does not extend to multiple pastes.  Finally, as anyone who uses a stylus or touch for input (as with a Tablet-PC), cut-and-paste becomes a real pain without CTL-x and CTL-v, for example.

So, can we use some of our thinking about chunking, visibility, and richness (extensibility of Move to multiple paste, for example) to come up with a new design?  One that might do a reasonable job of getting the best of both worlds, and be compatible with emerging pen and touch interfaces at the same time?

So, the approach that we came up with - significantly, by folowong this train of thought - was as follows:

- begin by using the traditional proof-reader's approach of circle and extend line to destination.

- create a temporary buffer (i.e., clipboard) by drawing a symbol in the margin of the page

- let that buffer be a legal destination to which something can be moved.

- to paste the buffer contents, draw it in the margin draw a line desired destination

- this enables moving things across page boundaries

- this enables multiple pastes

- by drawing different symbols, one has clear way of havng multiple self-defined buffers/clipboards

- by the symbols, one has a visual representation of clipboard(s)

- one could, for example, then double click on symbol to examine contents (which is consistent with GUI)

The key thing to note in the example is that it is builds upon skills that most users already had acquired, from a life-time of living in the everyday world, prior to ever using a computer.  Do these techniques render the existing GUI ones obsolete?  Not at all.  But what the example does do, I hope, is help illustrate the complexity of even seemingly simple transactions, and how some of these theories can aid us in thinking about them, and generating designs that may address some of our emerging UX challenges.

Sorry if I have run on a bit, and especially sorry if I missed your point.  But I tries

All the best.

Amazing! Thank you so much.  I really enjoyed the insights here.