GUI and NUI
by dovetailedlife
One of the cool things about going to school is that there always other stuff going on around you. Mostly for free, usually for really cheap. The other day I found out that Craig Mundie, Microsoft’s Chief Research and Strategy Officer was coming to demo the “new ways that humans will be interacting with computers with a 3D demo.”. I keep up with technology and some of the strides being made(and told myself that this had to be their new Kinect) so I decided that I had to be there.
After making the 15 minute trek from Duke Divinity to the Fuqua School of Business, I found my way(just in time I might add) to the auditorium that he was presenting in. If you aren’t a geek and don’t keep up with techie things, you’ll be interested to know that Microsoft Kinect (code named Project Natal) is a new way to interact with computers through the use of motions and gestures tracked by an incredible camera that not only recognizes your body, but can ignore unintended gestures as well. Think of it as the Wii without a controller.
The large point that Craig made was not one of the technology behind it (though it was pretty cool) but was more centered around the idea of why a device like this is needed.
When the original Macintosh was released, the breakthrough in the consumer market was one of the GUI, or graphical user interface. It was now possible to use a system of icons to convey a message, and made computers usable to the general public. You no longer needed to learn code in order to interact with the device. It was, in its truest form, revolutionary.
The point that Craig made was that the trend has moved from the GUI to NUI, or natural user interface. Does this system use graphics still? Of course. However, there is no longer a mouse and a keyboard, you tell the machine what to do by using voice commands and physical hand movements. In a way, it seems more…natural.
This occurred to me last night when Allie and I were out to dinner and observed a mother with several children. One child was in her lap, using her iPad to play a game. One was across the table with her iPhone, supposedly doing the same. The kids seemed, as far as I the creep across the restaurant could tell, to be able to entertain themselves VERY easily by just tapping on what they wanted to do. They saw the icon, they clicked it. If you think about it, it’s brilliant. They didn’t have to realize that funny shaped thing next to the computer moved a cursor. They didn’t have to find that cursor on the screen, move it around and then search a menu for what they wanted. They found the icon, tapped on it, and were off. Its like taking the graphical interface to the next level. It’s what Steve refers to as “magical” about the iPad.
This is the difference that Android and the iPhone have made in the mobile market. No longer did you have to scroll through menus with directional keys or navigate through menus with a ton of buttons or scroll wheels. Oh wait, I guess Android still requires that. No longer did you have to worry about having a stylus with you wherever you go (or losing it).
Sure, the Microsoft Kinect is more advanced in many ways than the interactions with the iPad, but it is the same concept. Perhaps computers don’t have to be so complicated. It should be relatively easy to do whatever you need to do, as quickly as possible.
It’s not perfect yet. Apple hasn’t quite seemed to figure out how to make it easy to manage lots of applications while maintaining the the simplicity. The new folders function seems to help, but isn’t perfect. The Microsoft Kinect works well (from the few minutes that I got to play with it) but the gestures have to be large and intentional in order to be recognized and consequently must often be repeated.
But. Imagine a world in the future when yo walk up to the table at a restaurant and the menu is a part of the table. You point to what you want and it expands to show you the options for preparation. That is already happening in man restaurants around the world with Microsoft’s Surface. Imagine never having to touch a cell phone while driving. Ever. And yet it can still be used to make calls hands free and navigate. That is already happening in many cars.
It’s changing our world as we know it. It will be interesting to see how it changes in the future. This is our world.
-B
There’s a fine line between simplifying user interfaces and dumbing them down. The former is good, as it makes them more accessible to a wider audience. The latter is bad, because while it might accomplish the same things in terms of accessibility to new users, it decreases usability for advanced users.
You joke about Android being too menu-based, but really the only difference between (say) the Froyo and iOS4 browsers (besides Flash support and the 50% difference in Javascript speed, snark snark) is that the Froyo browser has a “More” button that pops up a bunch of options that simply don’t exist on the iOS browser. Both are equally easy to use in every way and provide all the same features as long as you don’t go into that “More” menu. But when you want to, say, search a page for a term, on Android you can hit “More -> Find on Page” and you’re off. On iOS, you’re SOL without 3rd-party workarounds. (I have a friend with an iPad, and I’ve seen how this particular workaround works, too. It’s not what I would call “user-friendly.”)
That’s the difference between “more accessible” and “dumbing down”: what you can do. Advanced configuration menus can be daunting, but with a few reasonable defaults 95% of your user base will never have to go into them. But for the 5% of people who want to be able to get into the fine details of their phones/computers/what have you, they’re going to be sorely disappointed if those options just aren’t there.
I think the problem is that companies look at the astronomical sales figures of the iPad and the Wii and say “wow, this is clearly the future.” What they don’t see is that nobody uses their Wiis and that more Intel-based PCs are sold each week than the iPad will sell all year.
“Motions control and touchscreen will replace mice and keyboards in the next few years!” follows the same line of thinking that leads to “by the third trimester, there will be hundreds of babies inside you!” They’re novel, so you’re going to get a lot of “early adopter” sales, and they’re fine for working with a device that fits in the palm of your hand or playing with a digital tiger. For getting real work done, however, we’re a long way from replacing a good old fashioned keyboard and mouse.
And can you imagine trying to play StarCraft with just a Wiimote? It’d be awful.
There is no doubt that NUI designs are a long way off from taking over mice and keyboards. I actually think that Silicon Valley got it right when the decided to go with the GUI originally. That stuff is here to stay. But, both Gates and Jobs have said that the tablet is the way of the future. Moreso for the majority of the population.
I once had someone tell me that they would never own a Mac because they didn’t want a computer that was meant to be childlike in its approach.
If it wasn’t for the GUI, how would my grandparents send me all these ridiculous “the Muslims are taking over the world!” emails? But the lack of GUI would probably have meant that we would have never been able to video chat with each other from separate parts of the country either. We may have never used cell phones.
I think that direct input and manipulation without the need for a pointing device pulls back one more layer of complicating the computer user interface. THIS is Apple’s main goal.
And so, Apple was first to market with the phone that interacted in this way. Android has been able to learn from them and fix some of the things the left out. And it is always going to be a cat and mouse game of give and take. Apple chooses to put all of their settings in an app called “settings” instead of having a complicated menu system that the user would have to remember existed in order to be able to change anything. Because they were first and had no one to learn from, they have been behind in a few respects. We are agreed that the “find in page” function is missing. However, iOS 4.2 supports it. From the same search bar. Want to search, for whatever (google, recent, in page)? Type in the search bar. One place. No menu to navigate through.
I loved my G1 when I had it (most days). Because there were no closed APIs on Android, cooler stuff was developed for it faster. Which meant that heavier users had more stuff running. Which meant managing tasks closer. If not, the phone ran slowly and drained battery. How annoying. However, I loved that it copy and pasted. But I had a hard time selecting what I would like copied. Apple added copy and paste a lot later to the iPhone OS. But it was better.
Everyone knows that Apple’s system of notifications needs an overhaul too, BADLY. There is work yet to be done.
The best point to be made is probably that there is so much potential. Because the competition is so fierce, it means that development for both platforms is really healthy. This will enable NUI to become further developed and eventually take over in some neat ways.
And oh yeah, playing any game like StarCraft with your hands and arms alone (assuming that the camera isn’t at least 98% accurate) would suck.
Sent from my iPad