In this lecture, we're going to talk about designing an augmented reality. So now the virtual reality but an augmented reality so the whole point is that we need to start thinking about how to translate and transfer our knowledge from designing for virtual reality, how does it actually translate to augmented reality. There's a lot of things that are in common between AR and VR and so in this lecture, I assume that you know about VR so my previous lectures on VR design and we can then focus on what's really different and special with augmented reality. So We're going to have a case study here as well so our case study is done in unity, it's actually was an independent study that I did with my student who is now a PhD student with me, Shasta Rotherham, and we explored an AR vo education based simulation Kepler's Law of planetary motion and we actually visualized all three laws and we doing it with marker based AR so here on paper, you don't really see it exactly but, the paper functions as a marker. It also has information about the law and they're using it in very interesting ways so it's really a nice, I think, in my case, in my opinion is a nice use of marker based AR designed for tabletop scale and we also have a marker less version of it. So now I'm going to give you a quick overview of each of the projects showing you some of the interactions that we have implemented so in our case study, we're going to learn about marker based design with forea, here you see all three laws is the first law on paper and then we have using a smartphone. So we have, essentially you're holding the phone above the paper, and then you can interact with paper just like we did switch to a different law here and you can also then manipulate some of the parameters for each of the laws through the smartphone, so you actually have an interface on the smartphone and what I'm showing right now is actually what you see on the screen. You can tap the simulation to pause it for example, and then you can tap again to continue playing it. Keep in mind that the user is holding the phone above the paper and using the paper as a marker. And what is fascinating about microphase they are is actually how You have control very easy control over where the AR experience takes place the scale of the AR experience depending on how large your print the marker and how you design for the markers and it can actually be really cool like you can see how these planets are now floating on paper and I'm analyzing a little bit some of the alignment issues we have on paper but that's fine I mean, it is a still relatively robust market tracking solution, the foyer that we're using here and it works actually quite nicely. So we also going to learn about, a marker-less AR version using AR foundation. In the project interestingly we started with this version first but the way we're going to learn about AR in this course is through looking at marker-base first and then marker-less AR What you can see here, if you compare it in terms of the interaction design, there's no more paper now it's mica list so you can place it wherever I still placed it on the tabletop in my example, but then we had to think about how do we actually transition between the nos, you can't transition between paper anymore so we actually brought up this tabbed interface. And Shweta and I regret a little bit that vertical design of that tabular tabbed interface and but in retrospect in, retrospectively, we feel like we should have maybe designed this a little bit differently because we felt like it's not so easy to keep everything in view but then when I demo it like the way I demo it here doesn't it look cool, I think it's pretty cool. So we actually going to learn in this lecture about the differences in AR experiences, not just in this lecture, actually in this whole week focused on AR. So we're going to spend time on marker-based AR where the user actually uses a fiducial, like a physic often a physically printed out mocker on paper, and that marker is then the portal into the AR world, we're going to learn about marker less AR so using the device's a lens into the AR world and the device then does the hard work of actually doing environmental understanding and figuring out where it is establishing the 3d coordinate space so the system and the frame to really understand how it moves through 3d space. And we're also going to learn about by the first two at least in our case studies are focused on mobile AR and I think those are the ones that are very accessible to you if you have a smartphone that is AR capable supports arcade and AR core or just mockers using the webcam on the other rear camera on the smartphone we also going to learn about headworn ar. So I'm going to show you some examples of the HoloLens two, I realized that you may not have access to that device, but I still think it's very interesting because in the next few years, we can hope at least at the cost of a drop and you will be able to use head one AR so seeing the AR world directly and always and that is interesting. So think about all the evil episodes in black mirror if you've watched that show, yes anyway, so he has a quick overview of how marker based augmented reality actually works so you're putting up a marker, like this hero mocker here this is based on air to a kid and so it is a default fiducial you can train your own custom mockers, you can customize, you can even randomize in some of the toolkits out there. So before I, for example, which we're going to spend some time with you can actually generate macros and you always have this kind of trade off between planning with the environment and optimising for tracking. So these markers, for example, have to be designed in a specific way the black borders are important and we're going to learn about all this. And then what happens is, as the user brings in their smartphone here, like I'm showing this In a second, as the user brings in who has the phone above the marker or brings the marker into the view of the phone we actually then see a few interactions going on and we have the when the marker is found, we can visualize we can track it and we can visualize any content that was designed for that marker. So we show the marker conduct when it was found, when the marker was found and the market was lost, then normally we would hide the marker base content otherwise it'll look weird if you just leave it floating, unless you have a transition to, or less AR, which is called extended tracking. So switch from marker-based to marker less. And that's an option that you have if the user has a smartphone that supports for example, Elkhorn arcade so before you extended tracking is relatively easy to implement. So here's an example of me just doing just that so I'm actually using this slide and this is something that if you go to some of our examples, which I'm going to share with you, and then you can also just do that and try that out. We also have content where I'm going a little bit more into how to implement these kinds of things as part of the honors track so if you're really interested in doing this, I have some follow along lectures there in the first part of the main track, I'm going to keep it relatively Straightforward just focused on an overview. So we're also going to learn about maker-less augmented reality so maker-less augmented reality works a little differently, what you see here is actually visualized planes these were detected and we're going to learn about this in detail how surface or plane detection works. It's a very primitive form of environmental understanding, we learn about more advanced forms of environmental understanding as well, so environmental understanding is required to detect physical objects and place, then virtual objects in the environment but to learn about protection, we're going to learn about spatial sapping so spatial mapping usually uses some form of depth sensing. So you may have additional cameras for that you can to some extent be done with just a software, so software based augmented reality so AR core and AR kit are actually getting really good at doing spatial mapping as opposed to HoloLens which actually has real depth cameras. So we're going to learn a little bit about this and then the use of obviously, understanding the 3D geometry is that we can, for example, render occlusions so then we can make virtual kind of disappeared if it shouldn't be visible. And so if there isn't a physical object in the way, but here's a very simple example of using environmental understanding. And then there are some more advanced aspects like as I said occlusion or light estimation, like if you wanted that content to really fit in blend in nicely and not stand out like the way it does obviously it's an abstract shape, but it could be some kind of trophy or I don't know if somebody ever gives me a medal it could be an air medal. And then that will blend in there and if we actually adopt for the lighting conditions in this specific area so we need light estimation and light something, we're going to not touch on too much it's just something I want you to be aware of, so there are actually two things in for example in AR foundation that you can control for is actually on the specific platforms and AR core and aerokit so on smartphone based platforms are actually quite advanced there so they can control for ambient intensity so apply average lighting to an object and also color temperature and so actually use white buttons to color correct the object and I'm not really showing this in the screenshots so what is actually happening here? I'm going to show you quickly the demo so I'm just going to go here again to my desk, place the object this is one of the air foundation samples that I was just running here where I was proudly deploying it to my phone [LAUGH] quite an intense process to do this in unity and then run it and then you have basic interactions, this is part of the XR interactions toolkit is also now native integration in unity. And then you can like move things around scale rotate, and you have support for all that and the planes that are being attacked by constantly scanning the environment in the background that is the environmental understanding it's limited to horizontal and vertical surfaces at this stage and allow you to place these objects in the world. So we're going to learn about Mark-based and Mark-less we're going to compare it really what changes is in terms of the environment content and the interactions really change and so we're going to learn about this but at the highest level, obviously environment is limited when it comes to marker-based experiences, you actually don't actually have an understanding of the environment. [LAUGH] you're designing around that marker and the user can play that marker whatever they want to. Marker-less is obviously you're trying to get a read on the environment and that is interesting that allows you're more mobile you don't always have to keep a marker in view so It enables larger scale in terms of environment, then content I mean, something to think about is like how big should the content be appearing when it is relative to a marker and then when it's like in in the world rather than tabletop if you can also do marker-less tabletop experience. But it's more common to do marker-less and then room scale experiences and so that actually has an impact on the layout of the content release and then the scale and actually the amount of content you can show because from Marker-based they are at anytime the market needs to stay in view unless you have external tracking and that's when you start blurring these things. Interactions, you can obviously have more implicit interactions with the marker so, I mean, maybe the exquisite part is the user has to place it somewhere, but then the app just runs or you could have more explicit marker based interactions really asking the user to bring things into views or show specific parts of the interface so in our Kepler's Law example, we'll learn about switching between the laws actually by bringing in a different sheet of paper. And then for Maker-less AR, we also going to learn about differences in interactions obviously it could be touch base it could also be if it's head one AR, it could be gesture, and speech and so we're going to learn about these considerations as well. So think about this example, I'm going to show you throughout examples of Marker-based AR and so that's the tabletop experience that I was talking about, and then this is like the room scale and foundation, for example, so this is where AR are based and we're going to learn about this and I'm going to show you examples here as well so we have specific lectures focused on Market-based and Mark-less, and they are also going to learn about the difference between they are and had one AR. So I'm going to bring out my Hand-held, I'm going to show you a really cool well, relatively abstract, but cube example I built this relatively last minute I thought I had this idea of how to continue this thread of interactions throughout the lectures I hope you like it [LAUGH] It was not too difficult to implement using the mixed reality toolkit. Thanks to Microsoft was relatively easy to implement on the Holocene two, and here's my example and so we're going to move some cubes here and the cool thing now is because we have advanced environmental understanding these cubes and they have physics, so I gave them physics there are rigid body objects, and they use gravity and so then they actually can land on the world. You also just saw occlusion right this table here is actually occluding the virtual content, and I'm visualising the spatial mesh all the time the blue stuff here is the read of the device on the environment. And then I can do, we're going to learn about these interactions we're going to learn about near and far manipulation, both for VR and for AR. Here, this is AR, I was just doing a far manipulation using the cube and we're going to learn about all these things so it's going to be very cool and very interesting I'm looking forward to teaching you about AR We're also going to think about design issues, so this is really the way how throughout the exam very important to me, when it comes to AR, really we need to think about the social and the ethical concerns, they're significant ones because, these applications really depend on camera access, kind of like 24/7 camera access. Need to see anything and everything in order to make sense of the environment you know where I'm going with this, you get it, yes so there are significant concerns if you don't care, that's fine, but you are going to capture other people and they may actually care. And we have to think about at least since we cannot do too much about accessibility at the moment and equity is an issue smartphone based AR is available to a lot of people in the Western world. And but if either this whole lecture is based on HoloLens I was one of like how many people would actually I mean, you would probably enjoy it, but it would be disappointed because you couldn't do it for yourself unless you have access to such a device so equity and these devices are very expensive so equity is really an issue. And we're going to talk about that a little bit throughout the course, I really need to make sure that I mentioned these design issues because in the next few lectures, we're going to go deep on the technical and the development side and that's always when we forget to talk about these important higher level issues, including privacy and security. And this is something that I'll come back in week four so the last part of this MOOC, but really, in the first quarter is this part of the XM MOOC, that one that is supposed to lay the foundation. These are really important issues and so I just wanted to make sure again that I'll bring them up. I know as a little bit of a weakness, because we are so focused on the technology in this course, we don't have enough time to talk about these issues but please, in your design and development activities, we always consider these issues they should just like, they shouldn't be an afterthought they should be built into any and every decision you make as part of the development process you get it all right So I'm looking forward to teaching you more about AR Maccabaeus Michael is headboard.
