Google 6DOF videos review: move inside VR videos!

Around 40 days ago, I reported to you the news of Google presenting at SIGGRAPH 2020 a new algorithm for recording, encoding, and decoding 6DOF VR videos, that is movies inside which you can physically move. Today I managed to try its demo, so let me tell you how is it cool being able to move inside a video!

6DOF Videos

“360 videos are not VR”. This is a statement I’ve heard a bazillion times. And while I absolutely don’t agree on it, I agree on the fact that they offer a very limited VR experience: not only it is a completely passive one, but also one that doesn’t let you enjoy completely the world you are in because you have a fixed point of view.

6DOF Videos aim exactly at solving this problem: they are videos inside which you can move. For sure the experience is still passive, but it also can add the interactivity of letting you explore the scene you are seeing, of watching it from the point of view you prefer. They are much better than 3DOF Videos, but also present many more technological challenges: not only they are very hard to shoot, but they are also hard to transmit. Some years ago, I’ve watched a talk by Intel proposing a 6DOF video technology of its, but the resulting files were in the order of magnitude of terabytes per second, making it impossible to manage them properly. Now the technology has improved, but it is still too rough to be used properly.

Google Immersive LightField Video technology

If there is a company that knows one thing or two about video streaming, that one is Google. Youtube is the most popular video streaming service, and for sure Google wants to maintain this dominance even in the immersive field. That’s why Youtube VR is already available on various platforms and it is already able to stream 360 and 180 VR videos.

Google is of course also experimenting with 6DOF video and trying to solve their conundrum. Some months ago, it released an impressive demo called “Welcome to Light Fields” that let you enjoy 6DOF photos, that is 360 photos inside which you could move, enjoying the fact that the photo reacted as if you really moved inside the represented environment. It was impressive seeing the light reflections follow the movements of my head.

In August 2020, Google has presented at Siggraph a paper called “Immersive Light Field Video with a Layered Mesh Representation”, in which it presents a new workflow to record, encode and decode 6DOF videos. This is its abstract:

We present a system for capturing, reconstructing, compressing, and rendering high quality immersive light field video. We record immersive light fields using a custom array of 46 time-synchronized cameras distributed on the surface of a hemispherical, 92cm diameter dome. From this data we produce 6DOF volumetric videos with a wide 80-cm viewing baseline, 10 pixels per degree angular resolution, and a wide field of view (>220 degrees), at 30fps video frame rates. Even though the cameras are placed 18cm apart on average, our system can reconstruct objects as close as 20cm to the camera rig. We accomplish this by leveraging the recently introduced DeepView view interpolation algorithm, replacing its underlying multi-plane image (MPI) scene representation with a collection of spherical shells which are better suited for representing panoramic light field content. We further process this data to reduce the large number of shell layers to a small, fixed number of RGBA+depth layers without significant loss in visual quality. The resulting RGB, alpha, and depth channels in these layers are then compressed using conventional texture atlasing and video compression techniques. The final, compressed representation is lightweight and can be rendered on mobile VR/AR platforms or in a web browser.

and this is the presenting video (In my opinion, worth watching, because it explains well the algorithm behind it):

The cool thing that Google has made is that it has not only found a way to shoot 6DOF videos, but it has also developed an algorithm to encode and decode the videos efficiently. All the data of a single frame gets divided into multiple layers, and then the RGB+D data of these frames get compressed into a single texture. The final result is that every 6DOF frame becomes just an image frame, that so can be streamed using traditional video streaming technology. And the resulting required bandwidth is compatible with current fiber connections: 100-300Mbit/s. This means that Google could stream 6DOF videos using the already existing Youtube architecture and people could already enjoy 6DOF videos using the technology they already own (a browser and a standard connection)! That’s so cool.

But how is the resulting quality? Well, let me tell you in my review!

Google 6DOF Videos video review

Since I’m talking about a video technology, it seemed fair to me to make a video review! So here you are me trying the 6DOF videos sample application and reporting my first impression on it!

Google 6DOF Videos textual review

If you’re more into text, let me share my impressions with you here. After having enjoyed this simple technical demo, I think that Google is on the right path, but the technology is not ready for mainstream adoption yet.

The demo app is comprised by something like 15 videos, all very short, for a total duration of maybe 3-4 minutes, so it is very short, but the size of the downloadable ZIP file was around 8GB. The dimension of the player + the videos is so little compared to other 6DOF video experiments, but anyway it is quite big. The video player has been clearly made in Unity and it works through SteamVR.

As soon as you play it, you find yourself inside a very simple menu scene, through which you can access the videos. There are around 15 videos, and you can play them one after the other using a very simple UI that you have in the scene. If you have played “Welcome To The Ligthtfields”, you will find the UI pretty familiar.

As soon as the first video played, I could see a guy in a garage working on some mechanical stuff. I started moving, and actually I had not the usual sensation typical of 360 videos of the world being stuck to my face, but everything moved coherently. I would like to say “it felt like pure magic”, but actually it was pretty normal. What is impressive of VR, is that when it works, it is like reality, so you don’t even find it special if you don’t think about the underlying technology. I was moving inside a video, but it all felt so natural to me: I was with this guy, in his garage, and I was watching him doing stuff. And it felt good, with no weird sensations, no nausea… nothing. It was a bit like in real life. That’s the magic of 6DOF videos.

In all the upcoming videos, I had the same sensation: thanks to the good resolution of the videos and I fact that I could move in, they felt so natural, and from a technological standpoint it was great thinking that I was inside a video, but that it didn’t feel like a normal video. Google has made various examples to let you enjoy that the system works in many settings: various lighting conditions, objects that are close and distant from the camera, reflections on shiny surfaces, etc…

Anyway, the magic still falls shorts for various limitations, which make you understand that while this solution is cool, it is just a tech prototype showing what it may become in the future.

First of all, the videos are very short: like 20 seconds each. So it is more like looking at a GIF than a video. This makes you feel less attached and interested in the videos because there is not a plot to follow.

Then the real issue is that the moving area is very little: 80cm in diameter. 80 cm basically means that as soon as you move your head a bit more than a step in all directions, the visual becomes black and you can’t see anything anymore. It is really too small: the videos are nice and there is always something you would like to see better, like a face that an artist is painting or the face of a cute dog. But as soon as you move naturally to see them better, the system halts. You can’t really explore the scene, not even watching better the elements that are close to you. Even if the video is 6DOF, the only advantage that you really have is that you can move a bit your head without breaking the magic. But there is no way to explore what you have in front of you.

The compression algorithm works very well and it is impressive, but it is not without issues, too. I’ve tried the experience with my HTC Vive Cosmos, which has a very high-resolution display, and I could clearly spot that the resolution of the video was good, but not THAT GOOD. I could clearly spot its compression. For a comparison, Welcome To LightFields has a much better visual quality. For instance, I was disappointed by the scene of the sea in the distance because the sea waves looked too blurred.

Also, there were some visible artifacts in some scenes. In the scene of the artist painting the face of another girl, when I took a step to the left, I saw that the cheek of the girl was clearly divided into two and it looked quite creepy (even if it is Spooky October now). In the scene of the lake with the mountains in the background, I could see that there was some 3D noise. This was a quite new sensation to me: imagine the noise that you see when a video is in low quality and maybe also highly compressed… that colored white noise that blurs and ruins every frame. There was that one, but it was tri-dimensional, with its depth being closer than the object that I was seeing. Quite weird, if I have to tell you… it was the first time for me watching some 3D noise!

The last issue is that the videos were just 180 ones and not 360° ones. This is not a big deal, I guess it has been a choice, but anyway 180 is less immersive than 360.

Google 6DOF Videos final impressions

In the end, I think that this experiment by Google is just the first step of the long journey that will lead to the future 6DOF Youtube VR. It lets you understand the potential of the technology because it lets you move inside a video, a sensation that was new and incredible to me. But at the same time, the experience is too limited and shows too many little issues to be ready for the mass market: especially the area inside which you can move is too limited to be truly usable unless the viewer is seated (that probably is the initial target of 6DOF videos).

In any case, from a technological standpoint, this work is impressive. If you want to know more about it and also try the demo yourself, you can head out to its official page.