Augmented reality is anytime
we put virtual things into a real world environment.
Now the most common version of this is cellphone or mobile AR, right?
So this is, you have a cellphone and you point it at a surface,
and you see a virtual thing projected onto that surface.
That's the most common version, but
there are also things like the Microsoft HoloLens or the Magic Leap.
Which provide stereo versions of this where you look through stereo goggles and
they put a 3D virtual object into your world.
Microsoft in particular has started using the term mixed reality a lot, and
they use it to describe everyone of their headsets,
including both the HoloLens, and their fully enclosed HMDs.
And when they say mixed reality,
they're talking about everything up to and beyond normal augmented reality.
So in normal augmented reality, you see some of the real world.
And then there was this thing for a while called augmented virtuality,
that nobody likes using that term.
Where you would see your real hands in the virtual world.
This is something you saw in the Intel merged reality demo that they showed
a lot of places in 2017.
Microsoft's term for mixed reality, also involves
bringing virtual representations of real world objects, into a virtual space.
And that's why their fully enclosed HMDs are still under the mixed reality
umbrella.
So that means, for instance, I would put on a Microsoft HMD and I could see not
my actual desk but a virtual desk that was in the same location as my actual desk.
And a virtual computer that was in the same location as my actual computer.
That's their concept of mixed reality.
And you've seen this also in things like the Leap Motion, where Leap Motion for
a while would put their device on the front of an HMD.
And allow you to see your hands, virtual representations of your hands,
projected into that virtual space.
So that's what augmented reality and mixed reality, and virtual reality, and
extended reality with the XR all kind of meet.
There's an article by Patrick Catanzariti called What is Mixed Reality?
It's part of devdiner.com.
That you can look at to get a little bit more detail on all this information.
So there's three more topics we're going to talk about in our discussion
of AR today.
What is optimization, which is just critically important for any kind of AR,
particularly mobile AR.
The next is designing AR apps, things you want to keep in mind while doing that.
And then the third is really some best practices to play to the strength of AR
and minimize the weaknesses.
First off, let's talk about optimization.
Now, all of the things we mentioned for optimization for
VR are pretty much the same for AR,
especially because you're looking at a mobile platform a lot of the time.
One of the benefits of you get with AR is that usually you're only dealing with
a single viewpoint,
a single camera, rather than having to have the two eyes of virtual reality.
However, there are devices I mentioned, like the HoloLens and the Magic Leap
that do have stereo vision, and do still have to render two viewpoints.
However, you get the benefit of only have one viewpoint most of the time.
But, you also have the performance hit of tracking.
And that tracking takes a lot of time and a lot of processing.
There are cheap ways to do AR tracking,
where you just really pay attention to the orientation of the device, and
don't pay attention much to the actual environment.
What most AR now is going to a place where it's tracking
optically the environment around it and trying to pay attention to that.
So trying to pay attention to planes and surfaces and
point clouds in that kind of thing.
And that takes a lot of processing power.
So let's talk about some good features of how to design AR well.
The first thing is, you really want to focus on stickiness rather than novelty.
A novel AR app that's just like hey, look, it's an AR thing.
That's actually really cool initially but people fall off very quickly.
And in particular, when you're dealing with mobile apps, stickiness is very,
very helpful if you want to make any money off of it.
So, a good example of this is Pokemon GO.
Pokemon GO had AR.
The AR was one of the big selling points for it.
But honestly, the Pokemon experience was the real selling point and
is the real stickiness factor that kept people playing.
In fact, many, many people eventually turned off the AR aspect of the game
because it saved battery power and allowed them play for longer.
Another thing to think about,
a really critical question to think about is, how does AR make your app better?
If you're going to add it to an app,
what is it doing in the AR that you couldn't do otherwise?
If it's just taking more battery power to run an AR scene, but
it's basically the same experience.
That's not really adding much.
You want to have experiences where AR makes your app much better.
When adding AR to an app, you need to think about how it's, both,
an enticement and a barrier to entry.
So, AR's a cool new thing and people are going to want to check it out.
But also, it's something that people lack experience with.
And so, new users might find it fiddly or difficult to understand.
That being the case, you need to make sure that the AR aspect of it is pretty
simple in terms of what you're asking them to understand at any given point.
So if you do have a complex experience that's going to happen in AR,
give them something simple to start with.
And sort of scaffold that or ladder it so
that you're adding concepts on top of each other over time,
rather than dumping everything on somebody all at the same time.
Speaking of timing, it takes time for the AR to establish tracking of the space.
It takes time for it to understand where the planes and point clouds,
and things like that are.
So, you need to give the player an understanding of that.
A couple of ways to do this are, one, if it's still establishing tracking.
And it helps to move the device around as you're establishing tracking.
Tell the player this.
Tell them to kind of move the device around.
Give them a little animation,
kind of showing them how you want them to move the device.
And that will give them something to do, and
some feedback while it's establishing tracking.
Another thing to think about is, can you give them the opportunity to do something
before this tracking is really established well?
Can you allow them to kind of place something and
then refine that placement later once the tracking is established.
In terms of helping players understand AR, there's a few more things to think about.
The first is just this idea of privacy, you're asking players for
access to their camera on their device.
And so it's worthwhile to reassure them that you're not going to use that for
nefarious purposes.
Just make them feel a little more comfortable, and
a little more likely to use your app.
Another thing I strongly encourage you to do is to use animations to describe
interaction that you want players to use.
So if, for example, I'm going to encourage someone to rotate this cup in AR.
I could put up text to describe that, but then I have to translate that text into
multiple languages, and it might be confusing.
On the other hand, if I show an animation of an object and
a finger swiping across the object, and that object's rotating.
Then that will be much clearer and I don't have to translate it later.
Similarly, if placing an object don't expect the player to just sort of
tap on screen where they want the object to go.
Allow them to tap and hold, and
place a ghost of the object that they can drag around.
Because often, that first tap is not going to be terribly accurate.
But refinement with a ghost is actually a really easy thing for players to do.
Finally, let's talk about what to do when things go wrong, right?
When you lose tracking, for instance.
So definitely tell your player that you lost tracking,
don't just leave them hanging, right?
But it helps to tell them why, rather than just say that tracking was lost.
So, if it's because there's too little light, or there's too much movement in
the scene, or something like that, express that to the player.
However, in these descriptions, try to not use standard AR jargon,
like plane and point cloud, and feature.
Try to describe it in ways that are more natural English that people would
understand a little bit better.
If you do lose tracking, one thing you want to make sure to do is
hide the virtual objects until the tracking comes back.
Because as the tracking is reestablishing itself,
your objects might jitter or blink in and out of existence.
And you really don't want that.
Anytime you're asking for the player to interact,
you want to have a timer on the situation.
Because the longer they don't interact,
the more likely it is that they need you to tell them what to do.
And so, having a timer, and if a certain amount of time has expired,
giving them a prompt, or giving them a video showing them the kind of thing that
you want them to do could be really helpful.
