Visual-Tactile and Visual-Motor Synchrony

Loading...

From the course by University of London

Introduction to Virtual Reality

229 ratings

University of London

Introduction to Virtual Reality

229 ratings

Course 1 of 5 in the Specialization Virtual Reality

This course will introduce you to Virtual Reality (VR). The course will teach you everything from the basics of VR- the hardware and the history of VR- to different applications of VR, the psychology of Virtual Reality, and the challenges of the medium. The course is designed for people who are new to VR as a medium. You may have experienced some virtual reality before, and may have some hardware- but this course is suitable to individuals who have never experienced VR and those who do not have much hardware- we will explain Mobile VR as well as devices such as the Oculus Rift and HTC Vive. Introduction to Virtual Reality is the first course in the Virtual Reality Specialisation. A learner with no previous experience in Virtual Reality and/or game programming will be able to evaluate existing VR applications, and design, test, and implement their own VR experiences/games using Unity by the end of the specialisation.

From the lesson

The Psychology of VR: the Three Illusions

In this week, Professor Mel Slater will take us through the Three Illusions. We’ll discuss Place Illusion, Plausibility Illusion, and Embodiment Illusion, with some interesting tricks along the way!

The Pinocchio Illusion2:57

The Rubber Hand Illusion2:27

Psychological Effects of Embodiment Illusion7:59

Visual-Tactile and Visual-Motor Synchrony13:52

The Technical Setup of VR Embodiment2:23

Changes in Attitude, Behaviours, and Cognition9:50

Self-Counselling and Treating Depression5:01

Comparing CGI and 360 Video4:21

The Future of Embodiment in VR2:10

Meet the Instructors

Dr Sylvia Xueni Pan
Lecturer, Department of Computing
Goldsmiths, University of London
Dr Marco Gillies
Senior Lecturer
Computing Department, Goldsmiths, University of London

So Mel, I know you've been doing embodiment illusion in VR for many years now,

probably since 2008 or 2009.

Maybe you can tell us about what got you really started in this area.

Yes. So actually, it was earlier in around about 2004 or 2005.

I read a paper by Armel and Ramachandran,

who describes an experiment where they use the rubber hand illusion.

And just as we did in the demonstration before,

at the end of doing the rubber hand illusion,

they attacked the rubber hand.

And what they showed is that when you have synchronous visual tactile information,

so you are tapping the rubber hand at the same speed in places you are the real hand,

that under those conditions when you attack the rubber hand with a knife,

people's arousal goes up significantly.

Means their skin conductance,

the sweat level in their body goes up at the moment you try to attack with a knife.

And if you have asynchronous stimulation,

where you are tapping the rubber hand and

the real hand at different speeds in different places,

then the amount of arousal is much less.

And I got very interested in this from a virtual reality point of view,

because this feeling of having an illusion about your body

is kind of similar to what we were talking about before,

which is the illusion of presence,

the illusion of being in a place.

It's another kind of illusion,

it's not the same, but it's in the similar domain.

So I got very interested in this.

And then in 2005,

2006, I think it was,

I spent a year at the Institute of Neuroscience in Alicante

with Dr. Marie Sanchez Veevers,

where we set up a virtual reality lab there,

and we implemented the rubber hand illusion in virtual reality.

So in other words, instead of seeing a rubber hand on the table in front of you,

you saw a virtual arm coming out of your body.

And then we used the virtual reality wand,

which is a device which is tracked,

so the wand was tapping your real hand and what you

saw in virtual reality was a virtual ball tapping your virtual hand,

synchronously or asynchronously, we could program it either way.

And what we found, and this was published as a paper a bit later,

what we found is that basically you get exactly the same results

as Armel and Ramachandran got.

And also is what Navi Conkoing got in the very first paper that people

have this strong illusion that this is their hand,

the virtual hand is their real hand.

But then, because it's virtual reality,

we could go one step further.

So as we were saying before,

with the rubber hand illusion,

you're stuck with a rubber hand,

you can't do anything with the other and touch it.

But what we could do in virtual reality is that

after a few minutes of the synchronous tapping,

we could program the virtual arm to do some movements.

And what we're interested to see is,

even though we instructed the subjects to keep their real hand still,

we wanted to see if there was any response to the movements of their virtual hand.

And so what we did is we measured the amounts of

electrical muscle activity on their real hand.

And what we found is that during the period the virtual hand was doing this,

there was increased electrical activity in their real hand,

and that correlated with the subjective strength of the illusion.

So the more they had the illusion that the virtual arm was their arm,

the more that they would move when the virtual arm itself would move,

even though their real arm didn't move.

So, basically, that's how I got into the whole area of body illusions,

by trying to see if we could reproduce the rubber hand illusion in virtual.

Yeah. So basically it had little subtle muscle movement,

although they are trying to keep their hand as still as possible,

just because they saw something moving in front of them.

So if they had the asynchronous condition

and they saw their virtual hand rotate,

then they didn't have the same level of muscle movements.

And this is kind of triggered by something called visual tactile synchrony.

Yes.

Is there any other ways to induce body ownership in VR as you have experienced?

Yes. So this was, as you said,

visual tactile synchrony which is how the basic rubber hand illusion works.

But of course, in VR, you can be much more flexible.

So a couple of years later,

Dr. Sanchez Veevers, myself, Dr. Byrne Hotsbangrang,

Franco Takia from SSSA, and Pisa,

and Massimo Bergamasco, and some others,

we all joined together and we went to Italy for a few days.

And we programmed another version of the rubber hand illusion,

whereby you wore a data glove which tracks the movements of your hand.

And as you move your real hand,

the virtual hand that you saw,

you didn't see your real hand,

but as you moved your real hand,

the virtual hand that you saw would move either

the same, synchronously, or asynchronously.

And once again, we reproduced all the results of the rubber hand illusion,

that under the synchronous movement condition,

people would have the strong illusion that the virtual hand was their hand.

And also we got to what Harry Farmer earlier described called the proprioceptive drift,

also we got that in the other visual tactile version.

So we reproduced all the different aspects of the illusion.

So this is visual multi synchrony,

whereas the normal rubber hand illusion is done with visual tactile synchrony.

So clearly, once you move to virtual reality,

you can do things that you can't really do with physical props.

How about with all the other parts of the body?

Now we talk about the hand and maybe you can do the same with nose,

but then how about the rest of the body?

Yes. So the other thing you can do very easily in

virtual reality is deal with the whole body,

not just a body part.

So this was first done by Dr. Henry Curson at the Karolinska Institute.

And what he did was a kind of virtual reality.

What he did is he had a mannequin dummy,

a life-sized mannequin dummy made of plastic,

and he stuck two cameras on the dummy looking downwards.

So imagine there's a dummy here,

you see two cameras pointing downwards at the dummy.

So the cameras are pointing at the body of the dummy.

Meanwhile, the subject, who's somewhere else,

he wears or she wears

a head-mounted display and the head mounted display is fed from those two cameras.

So in other words, on the head mounted display,

the left lens is fed by the left camera

that's on top of the mannequin and the right lens is fed by the right camera.

So from the point of view of the subject,

when they put on the head mounted display and they look downwards,

they have to look down because there was no head tracking,

and they looked down, they would see,

instead of their real body,

they would see a mannequin body.

They would see this plastic body.

And then what the experimenter did is synchronously tap

the belly of the mannequin and the real belly of the subject synchronously.

So, if the subject is standing here and the mannequin is there,

the experimenter would be doing this,

tapping both bellies synchronously.

And now think about this from the point of view of the subject,

they're looking down, they see a mannequin body replacing their own.

They're seeing something tap it like this,

but of course, they're feeling the tapping because it's synchronously on their real body.

So what happens there is that you get the illusion the mannequin body is your body,

and if you stick a knife into the mannequin,

people respond with this arousal response,

that they feel it's an attack on their own body.

If you do the tapping asynchronously,

so you're tapping the real body and the mannequin body but not at the same time,

slightly different offsets of time and place,

then this illusion doesn't work to the same extent.

You don't get the same arousal when the body is attacked.

So in a sense, that's kind of similar to

the 360 degree video we've been talking about in this course,

but that's very different from model-based VR.

And can you kind of generate a similar illusion with model-based VR?

Yes. With model-based VR,

of course, just like with the rubber hand illusion,

we can go much further.

Because with model-based VR, you put on a head mounted display which is tracked.

So, first of all,

you can be looking anywhere.

And what happens when you look down towards

your body when you're wearing a head mounted display?

If you programmed it,

you can have a virtual body substituting your real body.

So you look down, you see your body,

and then we can do the visual tactile aspect because you could,

for example, tap the real body with a wand.

While that controls synchronously, a virtual object,

a ball or something, to tap the virtual body in the same place,

that could be anywhere on the body.

So you can do visual tactile.

The way we do this today is, for example,

we put some vibrators,

like simple phone type vibrates at the different parts of the body,

and then when a virtual object touches that part of your virtual body,

we fire the vibrator on your real body synchronously.

So, for example, you might see something touch

your virtual shoulder and that moment it touches the virtual shoulder,

we fire a vibrator on the virtual shoulder and that's

enough to give you the illusion that you've been touched.

And that visual tactile way of integrating

the virtual body into your body representation,

it feels like it's your body.

And then we can go one step further because we can track the body.

So we can put markers over the body and as you move,

we can program it so the virtual body moves the same.

So now you look down,

you see a virtual body,

you move your hand and the virtual hand moves the same.

So this is a really powerful clue that the virtual body is your body.

But the most important thing I want to say, first of all,

that it's kind of above both of those visual tactile and visual motor,

visual motor is when you move your body and you see the virtual body move synchronously,

it's simply the fact of looking down and seeing a body

substitute your own body is already enough to give you the illusion.

Of course, if you move and the virtual body doesn't move, then it breaks.

So if you want the person to move,

then you have to have this tracking, and so on.

But if they don't move and they just look down

and they see a virtual body, already that's enough.

This first person perspective view of a life-size virtual body visually

substituting your own is already enough to give you the illusion.

Very interesting. It just reminded me when I started working in VR,

I basically spent most of my time in

the cave-like system where you can actually see your body through the shattered glasses.

Real body.

Yes, I see my real body through the shattered glasses.

So there was never been a problem until HMD came out about 2013,

that's when I tried Oculus for the first time.

I tried the previous generations as well,

but with Oculus, because that's quite light-weight,

so I spent a bit longer time in it and I looked down and I realized nothing was there,

it was quite a shocking and weird experience for me.

But basically, what you're saying is we just put a body

of avatar there where I expect my body to be,

then that would give me some kind of illusion,

and if I happen to be also wearing a kind of full body suit where

I have maybe another avatar come here and tap on my shoulder and I feel the tap,

that would really also emphasize this illusion.

And if then my body is also tracked and I can see the animation in real time,

I should be able to do this in VR and that would

really give me the full-on embodiment illusion in VR.

Yes. Let me just go back over one thing you said,

you said that when the head mounted display came in 2013.

Actually what happened, how I got into this partly,

how I got into this whole field is in 1992.

I tried head mounted display, we had one,

I put it on and exactly like you said,

I looked down, I'm invisible.

It was really a shock.

A really profound shock.

So the first experiment we ever did in

virtual reality was to look at the effect of having a virtual body on presence.

This was the first thing we ever did,

it was published in 1993.

So we had two conditions in that experiment.

One was you look down, you're invisible.

The other one, you look down and you have a virtual body.

Of course, in those days in 1993,

the virtual body was very simple.

It was made of a few polygons but it was enough.

So, having a virtual body also increases your sense of presence,

your place illusion, your sense of being there

because how do we also know that we're here in the real world?

Well, our body integrates with the world.

As I'm looking around this room,

I see my hands, I see my body is part of the world.

If I was invisible,

I'd have like a kind of ghost feeling that I wasn't really here.

So, it's very important what you said about looking down and not seeing

a body also ties back to what we were

talking about before which is the sense of presence.

Exactly. It does give me the feeling of I'm actually disconnected from

this world because my body is not with me and therefore things are less relevant.

Yeah.

Okay, that's very interesting.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play