CES 2010: NUI with Bill Buxton

Play CES 2010: NUI with Bill Buxton

The Discussion

  • User profile image

    cool  stuff Smiley

    hoping for more natal info in CES... cant get more NUI than that Smiley


    what happend to the robbie bach vid though? Smiley

  • User profile image

    Great video!


    @aL_: They posted it a bit too early because in the interview they discuss the "Xbox Game Room" which isn't being announced until tonight at 6pm PST.

  • User profile image

    Bill, NUI, awesome.

  • User profile image

    i see..

    looks like kotaku picked it up though Smiley


  • User profile image

    A couple of points first, I’m not a “hater” of Win7 multitouch (must have hit a nerve there), but as Bill himself says, until you can understand what could be (and can be) wrong (invalid scenario usages, etc.) with a product, how can you get it right? I want touch to be a simpler, two way extension of communication with the computer. Multitouch is a great start, but it needs context. Like taking the step closer to someone changes the context of the interaction. (Side point: my hands were made to work towards each other, not down to a keyboard/mouse or out towards a screen) Second, I said that you would need “painters shoulders” if you are required to have your arms up and out in front of you with touch screens. (http://channel9.msdn.com/posts/LarryLarsen/A-Look-Behind-Mouse-20/)


    Bill makes an excellent point about reusing known or acquired skills. I agree that we shouldn’t limit touch to finger usage only as we’ve developed extensive skills with using the pen/pencil (stylus type) tools. These should be taken advantage of, rather than throw them out the new product door. I’ve often wished I could sketch out UI design, case scenarios, UML design, mind maps, etc. straight to the computer rather than having to manually (and awkwardly/un-naturally) transfer to the computer via mouse and keyboard. That’s why I like the concept of blend, but even the designers say the ability to draw would be handy. Right click for the digital pen.


    One thing I’d disagree with Bill is voice usage in motor vehicles. From the studies I’ve read and the first hand observations seen, once you get the brain to start the process of speech, the ability to focus on driving drops like you are under the influence of alcohol.


    I would love to see the design of the software that can recognise a device to provide help. Angles, light conditions, shapes, sizes, distance… That fuzzy logic could be very useful to a lot of future projects.

  • User profile image

    Great interview! "Sketching User Experiences" (guys it's "Experiences" not "Interfaces") was one of my favorite books from last year. It had a similar wake-up / call-to-action effect on me that I got the first time I read the .NET Framework Design Guidelines. I'd recommend Bill's book to anyone.

  • User profile image

    You sure he's a principle researcher, and not a principal?  Tongue Out

  • User profile image

    I purchased it awhile ago after seeing Bill's keynote at MIX.  The keynote was a nicely polished and exciting product, while the book goes all over the place.  Although I enjoyed the obsession circling around the orange juice machines and Bill's ability to trash Adobe while trying to make it seem something of a side note, I cannot say that I have taken anything tangible away from the material.  If I were to produce such a book, I would have printed one with many white pages intermixed with instructions to draw something on them.


    The beauty of design cannot be fully expressed by the use of unbounded creativity. It requires hard constraints that need to be addressed with due diligence.   If you want to see this in action, look no further than our own fossil record.  Nature provides great solutions to difficult problems and occasionally some awesome mistakes.

  • User profile image
    Bill Buxton

    A few quick comments on your post:


    Yes, "painter's shoulders" is an issue.  If we are working on a kiosk or a whiteboard, things may be fine.   In the former, the interaction will likely be brief, and in the latter, we can get close enough that the arms need not be extended out so far as to cause excess fatigue.  On the other hand, when sitting, and reaching over a keyboard, desk, etc. to a vertical monitor, we can generally expect less use and more fatigue, due to extent of reach.  There is a reason that drafting tables were in a sloping desk rather than vertical wall format!


    As for human skill, the interesting thing is that the essence of skill acquisition (and hence skill utilization and transfer) is the notion of "chunking".  (See  the following if you want more background:  http://billbuxton.com/chunking.html).  This prompts me to go beyond wondering if/when I should use touch or stylus, and rather - recognizing that I have two hands, and can use them together - consider when it is best to use them together:  for example, touch in my non-dominant hand, and stylus in my dominant one.  This reflects what we do in the everyday physical world, and provides a mechanism to "chunk" together low level tasks into a higher level one, using an existing skill.  We have to stop looking at technologies as being in competition.


    As for operating devices while driving, I don't think that we have a disagreement - however, your comment suggests to me that I should make the point more strongly than I did.  You will note that in the interview I did say minimizing cognitive interferance as well as visual and manual.  As well, I qualified things by saying something, "to the extent that it can be done safely."  But you are right, while having the perfect voice only phone (let's pretend that there could be such a thing) sitting on the passenger seat, and speaking to a remote person using it, it is definately NOT the same as having the same conversation with a live person sitting in the same seat under the same conditions.  The real person has cose to the same ability to perceive the world and context in which you are driving as you, so knows why you go silent, or when they need to shut up for a moment.  Hence, the driver's ability to flow in and out of the conversation with minimal distraction/overhead, is much better than it would be with the person on the phone, who as no idea that something was about to happen that really needed your full attention.  The reality is, that there are a large number of things that already distract us from our primary task of driving (tuning the radio, adjusting the heat, sipping on a soda, listening to an interview on the radio, thinking about your spouse ...).  So the real question is, where do we draw the line?  The related question is this:  with an understanding of the issues above, by worthy design, how low can we get the interference of a remote conversation?  Can we get it below the threshold of acceptable risk?  For example, would augmenting the audio with video that gives the remote person a better sence of context help?  I'm not advocating a design, or even suggesting that there is an acceptable solution.  The fact is, unless I ask such questions, I can't answer the question.  But you are absolutely right:  to even begin, we have to be aware of the issue, or we won't begin asking them.  What I do know is that it is certainly unacceptable to text, read email, search for an address, etc. by any combination of voice and touch while driving.  We both know that.  Thanks for the reminder.


    Thanks for the comments.



  • User profile image
    Bill Buxton


    Just one short comment. 


    I have no issues with your comments about my book - except for one:  in what way did I "trash Adobe"?  Why would I?  They are a great company - one very much worthy of respect.  That is why I used them as an example.  That is the same reason that I used Apple.   The message in the Adobe example was simple:  even outstanding and successful software companies have a poor record of developing new products in-house, and rely upon mergers and acquisitions for their growth.   Another reason that I used Adobe was that their data was available, and in a form  that readers could easilly check.  Finally, the argument that the example was intended to support was that if companies invested in a design process, then in-house development of new (as opposed to N+1) products would become another viable option for growth.  If there was criticism implied in the example, it applied to the software industry as a whole, not to any one company.  I thought that I made that clear.  If not, I apologize.

  • User profile image

    The great thing about stories is that everyone takes away something different.  From the Adobe story, the one line that stood out was on the bottom of page 70.  "But can you rely upon mergers and acquisitions as your sole strategy for growth?  Not in any company that I want to manage, work for, or invest in."  It seems to me that you nailed Adobe's corporate strategy right on the head.  In nowhere else in the book did you write about investment.  For Apple you did write about their terrible puck mouse, and with Dell's unfortunate foray into the MP3 player market.  When you ventured out of the design world and into the world of finance, specifically dealing with the future valuation of a partner or competitor, I found it worth noting.


    Perhaps "trash" was poor word choice.  However, I would not have considered an antonym as an alternative.  There isn’t a designer alive that doesn’t have a love-hate relationship with Adobe. 



    PS.  On the write-up on "chunking", has the use pervasive use of the copy and paste metaphor, over the last few decades, altered your perception of the burden imposed by un-chunking the move operation?  It seems to me that many of these micro-ops are in some way compiled by our brain, in such a way, as to hide the details involved.  For example typing, I do not think of the letters my fingers are pressing.  I just think of the words and my brain takes care of the rest.  The mind also does this when you read.  

    "Aoccdrnig to rsceearh at an elingsh uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer is in the rghit pclae. The rset can be a toatl mses and you can sitll raed it wouthit graet porbelm. Tihs is bcuseae we do not raed ervey lteter by itslef but the wrod as a wlohe."
  • User profile image

    Great video and discussion!

     Bill Buxton is an inspiration, and a very wise man.   I think it is important to ensure that applications and interfaces allow for flexibility and multi-modal interaction.  This is something that Bill Buxton has worked towards over his career.  It is not surprising that he has a background in music!

  • User profile image
    Bill Buxton

    Good question on chunking w.r.t. cut/copy & paste.  It helps tease out some interesting issues.  The interesting thing about human skill is that we can - through practice - transform from attentive problem solving behaviour to automatic skilled behaviour.  Once acquired, we can do remarkable things.  The question then reduces to, "At what cost?"  So let's look at the basic action, MOVE as an example.  As we all know, there are at least 2 ways to do this with a GUI.   (1) cut-and-paste, and drag-and-drop.   From the perspective of skill acquisition, a key challenge with cut-and-paste lies in the concept of the clip-board. It s this magic place which is (for most) invisible.  In contrast, with drag-and-drop, the whole transaction is visible and there is a real-world metaphor.  I think that it is pretty clear that the former is harder to learn and more prone to error for beginners.  However, with practice, cut-and-paste and the clipboard will become a highly learned skill, and therefore be automatic, not require concious attention during execution.  That is, when a highly learned skill, it will impose no higher demand on the end user than drag-and-drop.  While both enable one to effect a MOVE, the two techniques have important differences.  The prevue capability of drag-and-drop makes it especially approrpriate for some graphical tasks, for example.  But, there is also confusion whether drag-and-drop means move (yes, if on same disk) or copy (if from one disk to another) - a conceptual "bump" that will hit users, frequently after they think that they "get it".  Cut-and-paste is less visible, but having grokked the concept of the clipboard, one can exploit the skill to effectively "move" something from one location, to multiple locations.  That is, the added complexity of the clipboard means that the move operation can be extended to multiple pastes. 


    A quick summary up to here:  (1) even otherwise disconnected actions can chunk together to form a skill; (2) the UI design needs to consider impact on, and find a balance betwen user's acquisition of skill as well as performance once skill is acquired; (3) different ways to do the same thing, such as move, differ in the range of their semantics, and not just in how they are articulated.


    A key point of my chunking & phrasing paper is to say that through what I called "kinesthetic connectivity" we can accelerate the aggregation of sub-tasks into the desired skill.  However, that does not neccessarily rule out being able to still take advantage of the individual atomic skills that chunk to constitute the higher order aggregate skill.  Paste as a component of the higher level skill Move in the cut-and-paste-with-clipboard method, is an example of this.


    So far so good.  But up to now, I may seem to be avoiding your question, and stating the obvious.  So let's behave like designers and see how this approach to thinking about things might shed light on other ways to do things.  Can we augment current techniques, and/or find new and better techniques?


    In our work in my old Input Research Group at the University of Toronto, we (students Garry Hardock and Gord Kurtenbach and myself) looked at this very example.  So the one thing that leaps out from my description above, is the initial problem of the invisibility of the clipboard.  Not only are its contents visibile, rarely is there any effective explicit visible representation of the clipboard itself - something rare in the GUI.  On the other hand, there is the problem disadvantage of the drag-and-drop metaphor, in that it does not extend to multiple pastes.  Finally, as anyone who uses a stylus or touch for input (as with a Tablet-PC), cut-and-paste becomes a real pain without CTL-x and CTL-v, for example.


    So, can we use some of our thinking about chunking, visibility, and richness (extensibility of Move to multiple paste, for example) to come up with a new design?  One that might do a reasonable job of getting the best of both worlds, and be compatible with emerging pen and touch interfaces at the same time?


    So, the approach that we came up with - significantly, by folowong this train of thought - was as follows:


    - begin by using the traditional proof-reader's approach of circle and extend line to destination.

    - create a temporary buffer (i.e., clipboard) by drawing a symbol in the margin of the page

    - let that buffer be a legal destination to which something can be moved.

    - to paste the buffer contents, draw it in the margin draw a line desired destination

    - this enables moving things across page boundaries

    - this enables multiple pastes

    - by drawing different symbols, one has clear way of havng multiple self-defined buffers/clipboards

    - by the symbols, one has a visual representation of clipboard(s)

    - one could, for example, then double click on symbol to examine contents (which is consistent with GUI)


    The key thing to note in the example is that it is builds upon skills that most users already had acquired, from a life-time of living in the everyday world, prior to ever using a computer.  Do these techniques render the existing GUI ones obsolete?  Not at all.  But what the example does do, I hope, is help illustrate the complexity of even seemingly simple transactions, and how some of these theories can aid us in thinking about them, and generating designs that may address some of our emerging UX challenges.


    Sorry if I have run on a bit, and especially sorry if I missed your point.  But I tries Smiley


    All the best.

  • User profile image

    Amazing! Thank you so much.  I really enjoyed the insights here.

  • User profile image

    fyi: missing tag, "Bill Buxton", on video.

  • User profile image

    منتديات | فساتين

  • User profile image

    شات صوتي

  • User profile image

    بار بنات
    منتدى بار بنات

  • User profile image

    I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact, your creative writing abilities has inspired me to start my own blog now. Really the blogging is spreading its wings rapidly.

Add Your 2 Cents