In this video, we're going to discuss challenges in rendering realistic looking graphics in VR. In a lot of the applications we have talked about, it is overly important to create realistic looking graphical representations of the real world. For instance, in training and therapy, if the graphics do not reflect the real world, then the skills acquired in VR won't be transferable. In this field, you would really like to be able to represent the real world just like a photo. And that's why we use the term photorealistic to describe those virtual environments. Other times, we might want to create an imaginary fantasy world, but this does not mean that we no longer care about realism altogether. We still want the users to somehow understand and connect with that world. In order for this to happen, objects have to interact with lighting in a similar way as they do in real life. For instance, water ships do reflect lights in a certain way so that we perceive it as water. In order to create an environment that looks photorealistic or at least reflect to a certain degree how the real world works, we need to have a basic idea of how lighting works in real life. If you look around at the real objects around you, you can see that light interacts with different objects in different ways. Something made from fabric is a perfect diffuse surface, which reflects light evenly and equally in all directions, and produce a smooth and even looking reflection. On the other hand, something made from metal is a specular surface. In this case, lights are reflected towards a certain direction which produces a highlight effect. Diffuse reflection is computationally cheaper to produce. Specular reflection, on the other hand, is a bit more mathematically complicated, so we often say it is more expensive to render. But in everyday life, we usually have both of these extreme situations, and more often a combination of both. If we don't render any lighting effects in VR, things can look a bit flat rather than 3D. Here you can see now, I have added a light with diffuse reflection. This gives a smooth shading on the object and makes it look more three dimensional. Here, I have added specular lighting so you can see there are highlights. The most computationally expensive images to produce are glossy or mirror surfaces. These are surfaces you can see through or surfaces that reflect things around them. They're often considered to be extreme examples of specular reflection. These are all possible to render in VR, but normally at a very high cost. Next time you play a 3D computer game, pay attention to this. The chances are there won't be many examples of this type of surface. So, a quick summary. While you're planning to illuminate your VR environment, typically, there are three types of surface you will need to deal with: diffuse, specular and glossy or mirror. Diffuse surfaces are the cheapest to render. Specular are a little more complex, and glossy or mirror surfaces are very expensive. So, in VR or 3D computer graphics in general, when we talk about illumination realism, we mean how lights are reflected on the 3D objects. This is important in creating a realistic and believable environment. There are also practical functions. For example, architects may want to render their architectural design before building it for real, so that they can understand the lighting inside a building as well as the impact the building will have on surrounding areas. They might want to avoid creating dark areas, which can make people feel quite depressed and could even lead to a higher crime rate. Sometimes it is difficult to pin down exactly what an image is missing. In this case, B looks more real because it is an example of global illumination, whereas A is an example of local illumination. In local illumination, we only consider lights that come directly from the light source. In global illumination, we also consider the inter-reflection between objects. For instance, you can see that the lithosphere at the lower right corner looks slightly green because there is a green wall next to it. You may also have noticed that the back wall and the light on the ceiling have some more 3D details because of this. So, basically, global illumination creates more realistic environments, but it is more computationally expensive. That means that sometimes in order to render real-time graphics in VR while maintaining the required high frame rate, we will have to turn off global illumination. Now, these problems exist in 360 video because lighting there is recorded from reality, but you're essentially stuck with the lighting you captured. You can change it and represent it with different lighting conditions, with some post editing, but again because you don't have the 3D models, the editing you can do is very limited. You can't make the light change its direction and you can't change the reflective properties off the surface.
