In this segment, we're going to look at virtual reality technologies. Now, there is a lot of technologies out there, and what I want to do in this segment is actually look at some of the key ingredients. What does it take to create this immersion, this feeling of presence, some of the concepts we talked about earlier? What does it take in terms of technologies? We are going to look at virtual reality technologies and a few of the enabling ones from 360 photos and videos to stereoscopic displays, a rendering for each eye. We're also going to look at CAVEs, or CAVE, which is in a recursive acronym for Cave Automatic Virtual Environment. We're going to learn about three degrees of freedom versus six degrees of tracking on three DOF versus six DOF. We're going to learn about outside-in and inside-out tracking, the idea of putting sensors into the environment and actually embedding these sensors into the devices that we put on our body. We're going to learn about hand tracking, as I think that hand tracking is a technology that is rapidly improving and now becoming standard in a lot of VR devices. Finally, so not to just keep it entirely visual, I thought it is also important to incorporate spatial audio. All right. So now, let's look at these in a little bit more detail. What you can see here is that in this 360 video, I have agency, I have control. Because it was recorded in 360, it was captured all around the camera, and I can choose to look at Michael or not. Next, we're going to look at stereoscopic displays, so the concept of rendering a different view, a slightly different view for each eye. There is a little shift between the pupils, and that little shift then creates a depth view for our eyes. So it's actually quite cool. So here, we have the cardboard-vr-display source code. This has really a few demos that I think are cool. Here, we see an example of a rotating cube, and that is just 3D, okay? So we're just looking at it in 3D, but now, I'm going to start the stereoscopic view here, the stereoscopic display, and now, what it does is it creates a rendering for each eye, left eye, right eye. There is a tiny little shift between these eyes and this has to do with the interpupillary distance, the IPD, and that is something that can be adjusted in some headsets as well. So basically, we're not seeing two cursors and two cubes. Actually, having you put this on your face, you actually then perceive depth. You see one cursor. If it's done well, it gives you this illusion of depth, and that is the whole key behind stereoscopic displays. This is really the main thing behind cardboard. So remember, cardboard, phone goes inside and then we actually have a rendering for each eye with these barrel distortions. Left eye, right eye. Then we perceive it like this, like you see in the video, rendering for each eye. Cool. So that was that. I think that is really a first important thing. We're creating this immersive view, if you will, a rendering for each eye that gives you then a sense of depth, and we are starting to see things and we are starting to call this a virtual reality. 360 photos and videos by and in themselves are not virtual reality. They're often associated with virtual reality, but they only become a virtual reality when we see them through a stereoscopic viewer like the cardboard with a rendering for each eye. Because then our head movement can be tracked just the rotation, not normally the position. So we cannot actually walk inside 360 photos and videos, and we're starting to get control. We're gaining some agency over some of the content, at least in terms of rotation. My next example is a little bit different. It's a different display technology. It also creates a virtual reality display, but in this case, you're actually stepping inside what is called a Cave Automatic Virtual Environment inside a CAVE. Here, I'm showing it based on the example that a colleague of mine at the University of Michigan gave to me. Ted Hall, he is in our MIDEN, and he shows one of the applications he was working on for some of the researchers that allows him to explore a mammoth and look at it in 3D. What was really impressive here is how the view is rendered just for him. He wears these funny antennas and there's some tracking system, an OptiTrack System in the room that actually tracks these antennas and allows him to actually walk around. If you look at the video one more time, you will see how the view seems to be like somehow distorted and overlaid. That is the effect that is being created to give him a depth sense. So these will shift further away the further the virtual content is away from him and they will becoming more overlapping the closer the content is to him. When you go to the movies and watch a 3D movie, you put on your glasses that they give you, but then sometimes, you probably take them off because you're tired or not, or you just feel like, hey, let's look at this from a VR perspective. These 3D glasses are actually counteracting the shift and they're putting it together in front of you. There's usually some shadow display that can be active or passive, but some shadow display that then shows you just the image for the left eye and then the image for the right eye. Then this is such a high frequency that you don't even notice and it will create a 3D image for you that you perceive as depth. Let's look at actually, head tracking and also control tracking if you will. We have virtual reality motion controllers. They can be tracked in three degrees of freedom or six degrees of freedom. Let's look at this in a little bit more detail. Three degrees of freedom, like the cardboard, looks like this, and then we can track along the x, so pitch. We can track along y, yaw, but just rotation. We can track along z, but just again rotation, so roll. You cannot actually move along these axis but you can rotate your head. This is me in cardboard and this is pitch, yaw, and roll if you will. This is just three degrees of freedom, and if you can actually, also adjust your position. The second example here is the hollow kit, the one that I had earlier. It's a device wrapper just like cardboard, but a little different for AR. Phone goes inside, we can see through. If we expose the camera, we can actually do some tracking, and we can do positional tracking. Now, in addition to head, roll, pitch, and yaw, we can also actually, determine whether the device moves along the z-axis, along the x-axis, and along the y-axis. We can do this through tracking techniques like SLAM, which is simultaneous localization and mapping of the device in 3D space. These are some computer algorithms that work based off the camera feed, and allow us to determine some motion. Optical flow, if you will. Optical flow would be computing a vector as I move the device this way. We are producing a vector from here to there. Or if you will, what happens with the camera feed, all the pixels actually shift to the right. That way we can actually figure out that the device moved to the left. Now, that is what I'm showing here in this illustration. This has six degrees of freedom. We have also positional changes plus or minus x. Positional changes along the y-axis, and positional changes along the z-axis. Now, we do have position, and orientation, and we refer to this as six degrees of freedom tracking. Again, with this tracking, you can walk around in a 3D space. You can not just look around like with 360 video, but you can then move around. This gives you a lot of agency in the content. That's then when it starts to become hard for designers and developers that have to design for that agency. Here I have an example of the Oculus Go. The Oculus Go is technically a three degrees of freedom headset. It comes with the three degrees of freedom controller. The controller will always be on the right down there. We can figure out its rotation, but we cannot figure out where it actually moves in 3D space. It is really a little limited, but you can already do quite a bit just by being able to look around and rotate the controller. If we use this laser interaction here, this laser controller, laser pointer interaction, you can reach objects at a distance and you actually do have quite a bit of control already over the virtual reality content. Here we do have the Go. That's the controller I just showed, and that's the Go, the device that goes on your head. It is three degrees of freedom. No travel, just rotation, and the same for the controller. Just a little bit of a pointing device, but whether I have it here or there doesn't make a difference. Really, the difference is rotation. This makes really for quite some differences in terms of how you design and develop and then also how you can use virtual reality applications. We're going to learn more about this in other segments of the MOOC. When it comes to six degrees of freedom, a very good way to demonstrate this is through the Oculus Quest. You usually draw a boundary and then you can walk inside those bounds. That's then supposed to be like a safe zone. With six degrees of freedom, both for the controller and for the head, I can now walk around and move the controller in 3D space, and I'll be tracked all the time. That is really quite cool. This one has three degrees of freedom with a six degrees of freedom tracking for both the head and the controller. Now, we're going to talk about outside-in and inside-out tracking. In a nutshell, the difference between outside-in tracking is that you have some sensors in the environment that are pointed towards you and they track you as you move around in a certain area that is covered by these sensors. Inside-out tracking means you actually have those centers inbuilt into the device, the cameras are actually inside the device. I'm going to show you two examples here. My first example here for outside-in tracking is actually the original Oculus. The original Oculus required sensors to be placed on a table usually in front of you. These sensors here, were actually pointing towards you. Then they are creating a sensor field, and they can see you in this range and you had to make sure that these sensors are overlapping. There's some triangulation happening, and what they're actually looking for is light patterns emitted from the controller and also from the headset. My second example is the Oculus Rift S, which is basically the new version of the Rift. It actually has the cameras inbuilt into the device. Now, this makes it possible to actually do both the head-tracking and also tracking the controllers in six degrees of freedom without any external sensors. The cameras are on the device and basically this is the field of view that these cameras have. It's an approximation it's not absolutely correct here. You can see that the controllers are actually captured this way. In fact, the Oculus Rift S and the Quest even they also have additional cameras on the side. Even when you have the controllers next to the device or even below the device, the device can still see the controllers. This is actually quite an advancement over outside-in tracking. When you have these stands on the table, they could only see sensors, and usually the table would include things, so as you were doing things below the table you couldn't be tracked anymore. Now with inside-out tracking you actually have these sensors on the device and so anything that is visible, in a certain area around them can be tracked. When you put your hands behind your back, obviously this will not work, but that's fine. Here I have an example of the HTC Vive with a form of outside-in tracking. It's technically not absolutely correct, but this is what they usually looks like. You have a sensor in the environment and then the controller here is coming towards the sensor and it's being tracked in six degrees of freedom. Here's an example of me demonstrating their Rift S and I often do this. I'm pointing out the different cameras on the device and I am explaining how the Rift S can actually see. Has cameras on top, downwards to the left and right and front, and has a really good field of view for these cameras to actually track. I'm making some comparisons to the Oculus Rift original sensor stands that you see in the back. Let's talk about hand tracking. Hand tracking now has actually become a standard on the latest generation of VR devices. I'm going to introduce you to a hand tracking in two steps. First, I'll show you what this used to be like with a device called the Leap Motion, which I have here. The Leap Motion is a USB tethered device that would be mounted on top of the headset, and it creates a field of view, a window in which you can actually do hand tracking. I'm going to show you this first, and then I'm going to show you how hand tracking actually works on the Oculus Quest, which we have there. Now it's essentially inbuilt, so built into these devices. This is Leap Motion. It's really not very big and creates this window here. Then I can actually see my hands nicely articulated. In this example, I'm just moving my hands up and down and I can actually draw, if you will, with left and right and all the fingers. This is cool, first person view. This is from the app. Looking down. I'm fancying device on the side. This make sense. In this example I'm going to show Oculus hand tracking. Here under settings in the device, we do have hands and controllers, and you can see hand tracking and auto enabling hand tracking is actually toggled to on. Switched on. Now that means that I can just put down my controllers. Look at my hands, and there they are. I'm going to do pass through. You'll see what I'm looking at here on my hands and go back into VR and it's nicely mapped. It's relatively accurate. It feels pretty good. I am touching their correct fingers, and this would actually be a gesture, so lights up becomes active. What we're going to do is, well we can scroll and we can scroll sideways as well. I'm going to close this and can scroll sideways. It feels relatively good you can see the rendering of my hands. It still seems relatively accurate. Scroll up and down with this pinch to drag gesture. One thing that is a little bit, if an app doesn't support hand tracking like here, this is the browser. I can actually browse with hand tracking. Most normal websites would work, but then when you go and look at whether this now works in A-frame, then at this stage I have to disappoint you. I have to grab the controllers. We're going to do the passthrough, controllers over there. We're going to grab them, and we're going to go into the VR immersive view. There we go. There it is. Hand tracking and A-frame for default doesn't work yet, but this is probably going to happen very soon, just not at the time of this recording. I'm going to exit out with the Oculus button and this concludes my little demo of hand tracking and controllers. You can actually seamlessly go back and forth between hands and controllers. Pick up the controllers and press a button and there they are again and give it a little bit of rest and then look at your hands, give it a bit of time for the recognition to work. We do have hand tracking now built into these devices. I talked about VR gloves earlier and the reasons I don't have them, but the point of VR gloves is not necessary to tracking of the hands, it was actually the creation of the haptic sensation so that you actually feel like you're touching something. That can only be accomplished through some tricks by actually mapping the physical to the virtual environment so that for example, you are perceiving in VR a surface here and just happens to be also your desk that you are touching. If you can create these nice overlays between the physical and the virtual, you actually do get a sensation of physical feedback even without varying additional things like gloves and whatnot. Most of my examples were actually visual, so what does it take to create a virtual reality? Virtual reality is predominantly visual so in that sense it's not wrong. However, a very, very important part of creating immersion is actually the audio around the user, so my last example will focus on spatial audio. Here I could have chosen many examples. One that I like is actually Tilt Brush and I want to show it to you in two ways: first, without sound and then with sound. Now we have seen the main ingredients of Virtual Reality, both on the visual side and then on the tracking side, so stereoscopic views or alternatively creating a cave like monitor wall-size displays around the user, so you're inside a virtual reality headset, if you will, or for the headset that we've talked about stereoscopic views. We've learned a little bit about interactions and also hand tracking, and then spatial or positional audio as well. These together actually create a rich set of sensations and this is really what you want to accomplish so that your users feel immersed in Virtual Reality. The first thing's I'm going to revisit Beat Saber and we're going to look at it from the perspective of all these kinds of technologies that we've just learned about. In terms of the VR technologies that we have just learned about, Beat Saber really actually chooses those. It creates a stereoscopic display through the headset, it supports six degrees of freedom tracking which is really necessary. You need to be able to move in space, and also it does work actually with both outside-in and inside-out tracking, but the example that I'll show you here is using the Oculus Rift S with inside-out tracking. Down, down. I have to follow these directions when slicing. Oops, we should have used the other Saber, but that way you see what happens when things go wrong. Avoid those, and then just blue and now red. This one you can cut it into any direction. The synchronous cuts are fun, but I would like it when you get to slice like crazy like this. Now comes my favorite part of song. Up, up, any, any, down, down, down, right, down, down, left, down, down, right, down, down, any, any. We've made one mistake so far, but is okay. You're not pro. I have to be careful that I don't hit the cable or the case, because I don't want it to disconnect again. Down, and this one, I always find find it funny when they turn around and then you have to do the last minute changes. This one is cool. Slice, slice, slice, up..
