Before virtual reality became mainstream,
there was already a lot of interest in new ways of interacting, particularly in gaming.
The Nintendo Wii, the Sony Move and the
Microsoft Connect all tried to create new ways of interacting with games and computers.
They were less about fingers on buttons and more about moving our whole bodies.
They allowed us to interact by making gestures and to interact with the kinds of
movements that we would use
in the real world.
A great example of
this is the Nintendo Tennis game that used
a Wii controller and you play tennis simply by swinging your racket.
This kind of real world interaction
feels more real and it allows us to make use of the everyday skills that
we have learned simply
by existing in the world all our lives.
You don't have to know about gaming to play Wii Tennis.
You just have to know how to swing a tennis racket,
and not even that well.
So real-world interaction like this seems perfect for virtual reality
because all these technologies
lets us interact with
things as we would in the real world.
That is why so many VR headsets
come with similar interaction devices.
Sony PlayStation VR comes with the Sony Move.
The Oculus has Oculus Touch and the HTC Vive has its controllers.
But the so-called natural interaction techniques aren't necessarily better.
The great interaction designer Donald Norman wrote
an article critiquing these new interfaces or at least some of them.
Making a traditional user interface easy to use requires thinking about a lot of things.
The actions you can do need to be discoverable.
You need to be able to see what you can do in an interface.
In a traditional GUI,
you can see where the buttons are or the menus are,
and that shows you what you can do.
Interfaces need to be unambiguous.
It needs to be clear what will happen at any time.
You can't be in any doubt.
And they also need to give you feedback.
Once you've done something,
you need to see what effect has this had,
and you need to know that you've done the right thing.
Gestural interfaces often do none of these things.
It's hard to know what gestures you can do.
It's hard to know unambiguously which gesture does what.
And they often don't give you feedback on whether you're
doing a gesture correctly or not. Why is this?
If natural gestures are so natural,
why are they hard to use?
Well, sometimes they are not natural at all.
Sometimes there just isn't a natural action that corresponds in the real world.
If you're in VR editing a scene,
it's quite easy to imagine gestures for moving furniture around.
You just pick things up,
move them and put them down again.
But what about changing the color of an object?
It doesn't obviously correspond to a real-world action.
So you need a gesture,
but it might not be that natural.
And it's these artificial gestures that can be hard to use.
Even if there is a natural gesture to use,
it could be that your software just isn't that good at recognizing it.
For example, in a baseball game,
it's very natural to have a pitching gesture that is just like pitching in real life.
But that means your software needs to be able to tell exactly when you're making
a pitching gesture and when you're not
making a pitching gesture and just doing something else.
This can be really hard for software.
So sometimes you end up with gestures that are natural and like the real world,
but you have to do them in a certain particular way,
otherwise the software doesn't recognize it.
That can be really hard if you don't
guide people in how to perform the gesture correctly.
They can be doing the gesture and it feels right to them,
and they try again and again and again,
and the software just doesn't recognize it.
So real world body movement interaction
is really powerful particularly in virtual reality.
It can help us really feel that we're interacting with something real,
not just an interface.
So, you should make your virtual reality
interactions as close to the real world as possible.
But you also have to be careful.
You have to be aware when your interface doesn't
recognize every possible gesture all the time.
You have to help people discover what they can do and what they're supposed to do.
Provide visual guidance.
Make objects that you can interact with look different from those that you can't.
And you need to provide feedback so that people
know when they're doing a gesture or action correctly and when they're not.
That means people should see the impact of what they're doing immediately.
Things should move around or change just as your user is moving their hands.
Dr Sylvia Xueni PanLecturer, Department of Computing
Dr Marco GilliesSenior Lecturer
