Welcome back, in this lecture I'm delighted to have the opportunity to talk with Ben Shneiderman, one of the pioneers in the field of human-computer interaction and user interface design. And we're going to be talking about the ideas of direct manipulation and of user control. Ben is a professor at the University of Maryland where he founded the university's Human-Computer Interaction Lab that has become famous worldwide as a center for research in human-computer interaction. He's written a number of books including three of them that I've put up for you that are particularly notable. The Designing the User Interface book, Strategies for Effective Human-Computer Interaction, which recently came out in its sixth edition and is one of the leading textbooks in the field on user interface design. Perhaps my personal favorite though is Leonardo's Laptop: Human Needs and the New Computing Technologies, which is more a book about how do we take computing and use it to enable people to be the kind of creative and world changing intellects that we want in the modern era. And a recent one, The New ABC's of Research: Achieving Breakthrough Collaborations, for those thinking about how do we do serious innovative work in partnerships this is very much a must read. So, Ben, welcome to our class. So let's jump straight in and why don't we take, in your own words, what the idea behind direct manipulation is in user interface design and what it's meant to the field. >> Thank you, I'm very pleased to see how over many years direct manipulation has become an influential concept in the design of technologies, it's very much part of the Microsoft and the Apple user interface guidelines documents. The idea came to me as I was looking at the kind of applications that triggered enthusiasm from the users, as opposed to the kind of frustrated acceptance that many technology software products generated. And there were things like video games of course, air traffic control, some signs of simulations of scientific experiments. These all had the kind of fundamental property that it was a visual interface, that the objects of interest were visually presented on the screen. And then, the user could actually manipulate them, directly manipulate them. They would touch them with a touchscreen, or drag with a mouse, and they could move the objects from one place to another. They could open valves in a simulation, they could move players in a video game, they could select aircraft in an air traffic control environment. And those kind of exciting environments generate a sense of real enthusiasm among the users rather than reluctant acceptance. And so I began to look carefully at what it was that generated that kind of enthusiasm, and I wrote this down and began to formulate these principles. They were rapid, they were very responsive, and an arrow on slower technologies meant machines would take several seconds to respond to UNIX commands and other command line features, so the rapid reaction was important. They were incrementally, you could drag a little bit of an icon into a trash can or drag it back, you could resize a window, all those incremental changes were very effective. So rapid, incremental, and reversible, if you did something you could also undo something, you could reverse the action and that gave the sense of safety and a willingness to explore and be creative and so people were trying things out. Naturally, these blossomed out with the applications such as word processors that, at the time it was a great step forward to call them WYSIWYG, what you see is what you get. And putting a cursor in front of a letter and typing the backspace key would eliminate it, it was a remarkable change from the command line interfaces of the past. Similarly, painting programs became popular and we're talking now the early 80s, and the beginning arrival of new technologies with graphic user interfaces. And my contribution was to recognize the common features across these different applications and to describe them in a way that future designers could take these ideas and apply them to their own work. And so the characterization of the psychological aspects, the perceptual, the cognitive aspects was really the contribution, and that triggered a great cascade of innovative ideas from research labs as well as from companies. And we contributed things like the Home Finder application that allowed you to see a map of the Washington, D.C. area. You could move a marker for the place you worked, and another marker for the place your wife worked, and you could draw a circle around them. And you could look for the places that were in a good region that you could commutes to work easily. And that kind of interface which allowed you to have sliders for the number of bedrooms and for the cost, turned out to be fun and effective for those kind of applications. We built a variety of them and that inspired many others to keep going. [INAUDIBLE] then the touchscreen ideas that became so prevalent on mobile devices. We saw quite early on that that direct manipulation concept would be very effective if you could touch the screen directly with your finger, more direct than even a mouse or a light pen. And so In the late 90s we developed the idea of a touchscreen keyboard. First we had a 9 inch wide one and then we worked our way down a 3 inch wide one, so those small keyboards that you have on so many of the smartphones were derived from that idea which seemed remarkable at the time, of these 3 inch wide keyboards. Now to get that kind of precision out of a keyboard required some new thinking. Most keyboards were land on, when you touched the screen it would activate, our design change was liftoff. You put your finger on the screen and then you can rapidly, incrementally, and reversibly drag your finger back and forth and it activated when you lifted off. So there still are many discoveries and inventions to be made about improving the notion of direct manipulation. And to me I see great satisfaction that contemporary interfaces build on these principles and go even further with mobile devices, solid user interfaces, tangible user Interfaces, and those ideas continue to grow and spread. >> Well let's in fact take this into that long context. I'm sure some of our learners remember, but many of them I hope are too young to remember that the early computing that probably both of us did was as far from rapid as you could imagine. You would put that deck of punch cards in and come back perhaps the next day and get a response, and it was very much about data processing. If you were trying to run through a set of data and understand it, you would program up all of the things you wanted to compute on it, run a batch, and then get back a printout, which frankly made data exploration rather challenging. And one of the great innovations that drove the PCs, the Apple 2 in particular, was this creation of spreadsheets, of VisiCalc, of its competitors. And the idea that you could very quickly put something there and see what happened, and if you didn't like it, change it. I think there was always this other vision, maybe starting in science fiction but becoming reality, of systems where you wouldn't have to bother using the computer at all, you would just tell it what to do. And that vision still exists for some things but it feels like direct manipulation is in this middle space where people need to engage with what they're doing because it requires their thoughtfulness, their creativity, their ability to understand. And one of those areas that's been remarkably successful in your work and others has been visual data exploration. You've developed a system known as Spotfire and that was specifically about how do we very rapidly and interactively explore large sets of data. Could you say a little bit about how that's changed the way that people go out and think about analytics and visual exploration? >> Yes [COUGH] yes, we've come a long way Joe, the days of batch processing and decks of Fortran cards are long past. And that moved on quickly to command line interfaces where we could type commands and maybe wait 10 to 15 seconds and get a response, and that was a big step forward. But I think the graphic user interfaces that emerged in the 1980s in personal computers really made the difference you're talking about, they enable personal exploration of data. Our work on the direct manipulation, as you correctly point out, led to the commercial success story of Spotfire. I was pleased to develop initial designs and our paper at the Conference in 1994 led to the formation of the company in 1997, and so that was a big step forward. Many other visual Interaction companies have come forward, Tableau certainly is the big one these days, most widely used, terrific product. And there are lots of others that are growing in that space, and what that enables is the kind of creative exploration you describe. It's quite amazing to realize that even with datasets of million of items, we can have sub-second response time for selecting a region, for dragging an item, for moving a slider. And those allow for those creative exploration moments to do the what if exploration, to see what's happening. You're right in also pointing out an important step was VisiCalc, although it was tabular it was numeric, and data, and textual. It was not a graphic interface, but it had many of the properties of direct manipulation because you'd click on a cell and you'd change a value, and boom, the whole spreadsheet would update. And that was another thrilling step forward in the process of arriving at the modern interactive data exploration interfaces, so Spotfire was an important step forward. And another aspect of my creative process was the idea of tree maps, and you'd see the whole hard drive on the screen at once. Rather than typing the UNIX command of change directory, and list of files, [LAUGH] and move around one step at a time. To be able to see the entire hard drive was the goal of that project. And to me there's great satisfaction that there are hundreds of commercial applications and thousands of websites that use the tree map idea to show the entire economy, to show all the jobs in the US, show medical care. And these promote the kind of insights and understanding that I think is very important in our big data age when we are the beneficiaries of wonderful amounts of data of many kinds that enable us to better understand the complex world around us and support. So the direct manipulation concept influenced me in thinking about data in many ways. The Spotfire technology was part of that but then the tree map ideas was triggered by my desire to see the entire hard drive of computer on one screen without any scrolling. And so that led me to the strategy of showing the hierarchy of the hard drive and that concept of tree map has spread very widely. There are hundreds of commercial versions, thousands of websites that show the whole US economy, that show all the jobs, that show healthcare, that show many social, economic, and politics that enable people to understand better what's going in the complex world around us. And it enables them to make better decisions, that's what's really important. [INAUDIBLE] concepts are not just for big business and big government, they also work for personal needs. The quantified-self movement is building on the huge number of sensors that are available on our smart phones and the way we capture our lives. And we can better understand the health determinants to enable us to make better decisions about our lives, that's what really satisfying and important. >> So I have one more question for you because many of the folks who are taking these courses are thinking very much about mobile devices, they're thinking about developing apps. And in some ways when we think of computing power, when we think of screen size, processing apps are a big step backwards, that we have small computers and we have to make them usable. What's your take on how effective apps have been at employing direct manipulation principles? And is there any piece of advice or two you might offer for people thinking about designing their own apps? >> In general I think the million plus apps available through Android and iPhone stores are amazing. They generally demonstrate good visual design, the user interface guide encourages this direct manipulation principles. And standardized colors, and shapes, and positions of buttons have been pretty effective in getting good things out there. And you can take a look at recent success stories like Pokemon Go. What a great app, and what a great story, and what a great direct manipulation interface to let you move around in the world and throw the balls at the Pokey characters, and those show the vibrancy of effective design. Yes, I've seen lots of apps that have poor design where the touchable buttons vary in size, and shape, and color, and they lead to some confusion. But by and large those apps are pretty good, now the dilemma is we have so many apps that we have to keep track of them. And so I think we're going to see a coalition of app makers and larger platforms of apps that let you have some consistency so that all of your healthcare applications might have a common strategy. And so you could share your data from your blood pressure machine, from your thermometer, from your scale, from all those devices that give us the information about how our body is doing. >> Well wonderful, well thank you so much for joining us with a message that is not only informative, but uplifting about the fact that we've really succeeded in this field in making a difference and have a bright future to a large extent because we've thought about and now put into practice ways of putting the user in control. >> Let me add to that, that I'm very proud that our field of human-computer interaction has really devoted itself to the notions of societal transformation. We've had powerful effects and increasingly we understand that our role, for example in the United Nations sustainability development goals can be very influential. We can make the world a better place, we have made the world a better place, and we will continue to do so. >> Well, that's a wonderful way to end. This has been an interview with Ben Shneiderman on direct manipulation and user control. We'll see you next time.
