Issues (and advices) in shooting VR-videos and 360-videos

As a shower thought of the day, I was thinking about 360-videos. We all love them, especially if they’re 3D and if they tell an interesting story (like Ashes To Ashes does). There’s a reason if “storytelling” is one of the buzzwords of VR ecosystem: virtual reality is a great medium to create empathy, to create strong feelings by living stories. And 360-videos are a good medium for offering high-quality storytelling.

But shooting 360 videos has lots of issues and here I want to highlight some of them.

Hardware

How to shoot this kind of videos? There are various means, but the most popular are:

• 360-cameras: for €249 you can afford buying a Samsung Gear 360 (2017 edition), which can record 360-videos up to 4K in resolution! Really cool. But… these videos will not be 3D, meaning that you will see the videos as projected on a sphere around you, you won’t get the sense of depth of the objects contained in it;
• 2+ GoPro: as PornFoxVR guy told me, with 2 GoPro cameras and some post-processing you can record a 3D video with a 180° FOV. Incrementing the number of GoPros and using dedicated programs, you can shoot 3D videos with a bigger field of view. This solution is more expensive than the previous one, but can offer 3D imagery that in my opinion makes the video more real, more VR (some people argue that 360-videos are not true VR);
• Professional VR cameras: professional spheres full of cameras that record a 360-video in 3D. These are the devices used by Jaunt or by other professional companies. Nokia sells a camera of this type for $45.000, so we’re talking about professional stuff;

• Lytro-like cameras: super-enormous and super-expensive stuff dedicated in taking a volumetric 360-3D-video. With this I mean that the camera + the software try to reconstruct everything about the scene around it, so that the user can also move the head while watching the video, as if he were really inside the scene. This solutions are still experimental, but are very promising and offer what we really want by a video seen in VR: feel like being really there.

https://vimeo.com/206480546

Most 360 videos are recorded with the first kind of devices, since they’re cheap and offer a good quality. The more you spend, the more you get sense of immersion, of course. This means that at present time most videos appear as flat and don’t make the user feel really like inside the scene. Furthermore resolution is often far than optimal and again, this breaks the immersion.

Professional videos (like Jaunt ones), instead, gave me some WOW moments, but we’ll have to wait a lot before these will be publicly affordable. I’ve not been able to try Lytro videos, so I can’t express a personal feedbacks about them.

Distortions

If  you have ever looked these kind of videos, you have surely noticed that there are lots of distortions. In 180 videos, there are often distorted imagery near the view boundaries. With 360 videos, you can notice them on the top and bottom of the sphere and sometimes at the stiching point.

To reduce distortion issues, use high-quality cameras and software. These will not prevent distortion from happening, but will reduce them whenever possible.

Drawing Attention

In traditional movies, the camera points you to where you have to look to follow the story: if you’re looking a romantic movie for instance (why? Better look Dodgeball!), the frames show you the man and the woman that are talking together, plus some useless background.

But if the video is 360, you just look the environment as if you were in the scene, so the dialogue may happen in front of you, but you may be distracted in seeing a squirrel in the park behind you that is eating a nut and that has been recorded by chance. With 360-videos is hard to guarantee that the user will see all the most important parts of the story, to guarantee that he will follow all the plot. So, how to draw user’s attention to what he has to see? Here there are some ideas:

• Make all the important actions happen in the same position. In Ashes to Ashes this has been widely employed: all things that are meant to be seen happen basically in front of the user. If you want to rotate your head and look the surroundings, you can, but you do that at your own risk: you know that you may lose something important of the plot;

• Use sound: if you hear an explosion behind you (thanks to 360-audio), you will automatically rotate your head. So, if you want to draw attention, just use sound cues: in the example above of the romantic movie, hearing people arguing loudly in front of me would draw my attention there;
• Use other cues: maybe if I’m looking at the wrong direction, a girl in the video may point me to the part I’ve to look saying to me: “Hey, look! What is that?”.

With 180-degrees videos, this problem is highly reduced, but at the cost of reduced immersion.

A final thought on how to use this at your advantage: some videos show a main story and then various secondary stories that the user can follow only if he rotate his head from the main events: this way every user needs to see the video a lot of times to appreciate it completely and this increases its durability. Dear Angelica, for example, has a mechanism similar to this one… and at the end it advices the user to see it again to appreciate every detail of every single scene.

Framing language

This is one of the biggest problems in 360-videos, in my opinion. With traditional filmography, we have “framing”. The director may decide to zoom an a detail, then to move the camera away and show you a landscape and so on. Some movies have an amazing framing that it itself can convey you emotions… some games have that too: I think about Resident Evil games, where the camera was always put so that you had difficulties in seeing all the scene, on knowing what monster was expecting you behind a corner.

Playing with cameras is a powerful tool that directors have in their hands. We all remember awesome scenes of western movies where before a duel we could see a close-up of the two cow-boys watching each other in those eternal moments before firing.

In 360 videos this does not exist. And maybe will never exist. Since 360-videos emulate the fact that you are there in the scene looking with your eyes, you can not zoom in, out or doing something like that. There’s only one point of view, static, around which all events happen. There can be multiple scenes, with multiple points of view, but every scene is static. On the long time this results boring, flat. There are no more emotions conveyed by framing, because it does not exist in real life with our eyes. This problem is made even worse by motion sickness issues.

Motion sickness

The first time I discovered motion sickness in 360-videos has been when I made my friend Valentina to try my brand new GearVR. When she tried the video about Iceland, she got sick because there was a scene shot from a moving helicopter. I was surprised, but then I understood it was perfectly normal (even because women are more prone to sickness)… and I myself experienced sickness while looking vidoes like Ghost In The Shell Experience.

As I’ve already explained here, you got sickness when you stay still but the virtual images show you that you’re moving… and 360 videos are not an exception. So, if you have to shoot a 360-video, you have to keep your camera fixed. Again, this means a boring scene where you’re fixed in a point. There’s no way to move the camera, to make a shooting where you move around all the scene and make see it while moving: people would just puke watching the video. If you make the main character to walk, again, you trigger sickness.

This problem is even worse if you put an action camera on the head of the actor: his movement, his wobbling of the head, etc… would cause motion sickness in users. That’s why I’m not a huge fan of 360-action-cameras: how could I watch an immersive video of a guy jumping down a mountain without puking? And the problem is the same for game-streaming (like Twitch): how can I watch another player moving in VR without feeling sick?

You may say that this is the same problems of VR games and yes that’s true… but here we have one sickness factor that we can’t remove: the movement is always passive. Game guidelines suggest you that you have to move camera only when the movement has been authorized by the user (e.g. pressing an arrow key): the video is a passive medium, so movement is always un-authorized.

How to solve this issue? Well, I think that can be applied almost the same solutions of games:

• Keep the camera stationary: it’s banal, but it’s the safest solution. No sickness is possible this way;
• Reduce FOV while moving: 360-video-encoding could have some triggers to order the VR-video-player to reduce the FOV of the user while there is movement: this would make the experience a little less immersive, but a bit more comfortable;
• Keep camera speed constant. Moving at a constant speed causes almost zero-sickness… accelerations create sickness. So, if 360-video-directors could have some speed stabilizing devices, cameras could move at a fixed constant speed in movies and movement could cause less sickness.

As a last thing, I have to remember that 360-videos do not offer positional tracking (unless you use the innovative systems that I cited in the first paragraph), so you may move in the physical world and see your point of view of the video to stay fixed and this kills immersion and gives sickness again.

Position

You have to take care of where to put the camera to not break the presence and to give realism. There was an awesome video on Youtube explaining this (of course, I’ve lost the link!): you have to put the camera where the user would expect to have his/her head in that situation. If you put the 360 camera on the table to show people that are having dinner… well, that’s the point of view of food, not of people dining! So, again, put camera where user would expect to have his head and his eyes.

This can also be a powerful tool, that can substitute in part framing (well, positioning the camera is actually part of the framing process itself…). Putting yourself in an unconfortable position can make you change the sensations you get from the scene: seeing yourself inside a plate on a table could give you discomfort and that can be exactly the sensation the director wants to convey.

Emotions

Remember always that some images seen in VR could give strong sensations in your users, so be careful on what you show.

Sometimes normal events shown in 2D have no effect, while can appear disturbing while watched in VR. Like this footage of a monkey stealing a 360-camera… if you look the video while in VR, well, it’s quite disturbing.
https://static01.nyt.com/video/players/offsite/index.html?videoId=100000004735472

External elements

Shooting in 360 means that the scene should be all clear. Nowadays on a Hollywood set there are lots of people, lots of cameras, lots of tools around. If you shoot 360-degrees, you see all the scene, so there can not be any shooting devices or part of the crew there and this is so hard to obtain. When I watched Ashes To Ashes, I could clearly see some pieces of the filming rig below me and this again broke presence.

Furthermore, even the director should not be inside the scene to guide people. All of this is incredible difficult to obtain.

Fatigue

Remember that the user most probably is on a chair, at rest. If you continue to make his head to rotate back and forth to follow all different part of the action, most probably he would have to go to a physiotherapist to have relief for the neck strain. So, again, the video has to be crafted so that the most important events happen only in one direction (or maybe two).

Honestly, due to all these issues, I don’t believe that VR videos will become the new standard of filmography, I don’t believe that they will supersede traditional filmography. I think that they will be an additional medium to tell stories: a medium that conveys emotions, a medium that can make the user more immersed inside the scene. A medium that has its problems and its strength, like all other media of communications. We have still to find at first how to suitably communicate with this medium (find all methods of recording the movies, what solutions can be found to convey different emotional status in the users, how to draw attention, etc…); then we’ll able to understand for which kind of experiences this medium is more suitable (long movies or short movies; action scenes or just exploration; journalism or cinema movies; etc…). I’m very curious about the future, of course.

If you have any opinions, as always, tell me on twitter or here in the comments!