Magic Leap’s John Gaeta talks about the future of cinema, storytelling and mixed reality

I would like to ask you when and where you started working in the movie scene. Is it true that you started working on Saturday Night Live? (0:07)

Yes, it is true: it was a spectacular job: I was involved in all the short movies that they made. This introduced me to lots of interesting people and taught me a lot about fast writing and production of short comic movies.

At the same time I was living in Greenwich Village and I had some talented friends working in the advertising sector, making things like stop-motion animation for advertising… and these friends told me that a new studio was being built in Massachusetts by Douglas Trumbull. Douglas went away from Hollywood to create this super-studio with all the cutting-edge technology (like robotic camera systems, computers, etc…). All sort of innovations were happening in that studio, so my group of friends pulled me there and I got a job in the camera department. Within a year, I was already shooting scenes with a running camera.

Douglas was not only focused on movies but on immersive media, in creating the future of cinema and he invented a lot in this sense (like the Imax format). He was obsessed with Virtual Reality.

He was like a mentor for me, he taught that if something doesn’t exist, you can make it yourself.
When he took the Imax route, he left us in his studio, but without him anymore, and we asked ourselves what to do. We did two movies: one was What Dreams May Come, that has won an Academy Award, and the other one was The Matrix. And in three-four months, our small team made both the painting effect of What Dreams May Come and the bullet-time effect of The Matrix.

For Robin Williams, there was a fantastic look, fantastic special effects… (6:34)

Wonderful. One of the colleagues of mine of that time was Kim Libreri, that is now the CTO of Epic Games and he created the optical flow for the paint effect for What Dreams May Come.

For the Matrix trilogy it was important the use of computer vision, immersive photography, volumetric capture, the first markerless motion capture, to create frame by frame models of the performances… and today all these techniques are prolific in virtual reality and mixed reality. And all these things connected all different kind of people there, and we worked together with different skills on the same things.

After The Matrix, you made Speed Racer. Can you talk about that movie? (8:23)

It was one of my favorite projects. We thought that it was more important to work on how the movie could make you feel than in making it seem real, so everything was developed so to be artistic and expressive. That movie has an interesting combination of 3D, 2.5D and 2D mixed together. We went to Europe and found some amazing locations for the movie.

I am more an idea person, I like to work with talented engineers and artists… but you have to create such an amazing team by hand-picking them. The team has to be small and be able to work both on the creative and technical side. With Speed Racer, the team was a very high-level visualization team.

How has Matrix changed the perception of the audience at the cinema? After that movie, do all people want always to see that kind of cool special effects? (11:58)

I have to say, I don’t care really much about special effects, I’m not interested in special-effect movies, unless they’re interesting, unless they tell interesting stories. Matrix resonated for a couple of reasons:

1. It came out in the 90s, and it was the period that the average people were accepting the Internet. It took time for people to accept the Internet and not fear it anymore. The movie came when people started thinking about a digital world where we are together (the Internet), where we were going beyond the inflection point of mainstream adoption of the Internet. The science fiction leap was “You are inside the internet now. The virtual reality means being in the Internet”. Such revolutions do not happen often, and now this is happening again, with VR, AR… but regarding immersive technologies, apart from maybe Pokemon Go, we have not seen something mainstream up to now, we are not at the inflection point.
   In the last 5 years, I’ve seen another movie having such an impact, but not in the mainstream audience, but in the technology sector: “Her” by Spike Jonze. He figured out that Artificial Intelligence would have been humanized at some point, in a period when people didn’t even know what AI was. It resonated because techies were experimenting with AI interfaces, and the movie made these people understand that someone may also fall in love with these AIs. That rippled in many Silicon Valley labs, and made people think “Uhm, how can I get there?”.
2. The early bullet-time effect was something a bit like quick-time-VR, but a much higher level… you know, it is impossible to cheat time and space with a camera. People knew it was not possible to do that: the film resonated because only in a simulation, only in virtual reality you can cheat time and space. In 5 years or such, in VR we will be able to do that every time we want.

At Lucasfilm we made an incredible high-resolution high-fidelity very-close-to-reality looking tests… and you may have seen Mica, the AI human that Magic Leap put out… this is the best virtual human that I’ve seen and it is actually real-time.

Matrix resonated because the directors were openminded and allowed us to use the techniques that we wanted. They were ok in letting us experimenting and taking risks. Today the movie industry is a zero-risk one. And so we tried techniques that we thought were ideal for actual virtual reality, we captured the world both from inward and from outward. Thanks to hard work and luck we managed to do a great job in that. It is possible for people to create these innovative things in AR, VR, movies if they are allowed to innovate.

Why do people need VR? (20:38)

I think people will want and need Mixed Reality. Virtual Reality can be transcendent and transformational for people because it disconnects you and it can change people… but I’m not sure we need it. It’s definitely a choice.

But Mixed Reality… there are amazing things that can be made… but as every medium it has good and bad sides… even radio, TV (it could be great for education… oh no, it’s a propaganda tool) or Internet (place of knowledge, and in the end it is full of hate and distortion). VR can take you to a spectacular place you will never be able to go, but Mixed Reality can be prolific, it has the potential to go mainstream, it has the ability to make people come together, something that in VR is difficult to do, because VR encloses you in a headset and so you don’t see people surrounding you and the place around you… but it is a great expression and artistic medium, maybe like no one other ever. Mixed reality preserve the opportunity to gather together, collaborate together, educate, in a very powerful way, also because the application can have a powerful backend in the cloud, with remarkable processing power. But the key is people being together. VR is not, but mixed reality is going to be the next medium beyond television, that will be everywhere.

And will we see some kind of VR viewer inside the cinemas in the future? (24:27)

I worked on that for years, it was one of my obsessions. We make movies and movies are very technically driven, and as real-time computer graphics will start having a quality comparable to the one of slow-rendering one, we’ll start seeing lots of content produced inside engines, game engines. The world “game engine” has to be changed in “real-time graphics engines” and these will be the place where we will have volumetric photography, photorealistic scenes, etc… All the scenes will be produced in those engines and so the scenes will be of course exported for the cinema, but you can offer people the opportunity to pick a scene and let them teleport inside it so that they are inside the scene… I’ve done tests on this for years at Lucasfilms … you can return back to a particular scene and enter there and move inside there. Movies can have portals.

Is this something that ILMxLAB is working on? (26:47)

Yes, let’s see how they will do… but also others are working on it.

Once you have the cinema rendered in a real-time engine, you can do a lot of things with it: you can enter into scenes, you can project it holographically, as like in an immersive CAVE, or using glasses. You can have this immersive experience where you see the spaceships of Star Wars dropping droids around you… and this could use exactly the same assets from the film: it is not a copy, a remake… it is not the movie made by a company and the XR experience made by an external game studio that has not participated in the making of the movie. It’s the same crew that can do both.

We talked about VR storytelling and there are cool experiments with it, but I have not seen yet compelling storytelling content in AR and MR. Since you are inside Magic Leap… what are you doing to make compelling mixed reality content and what is the future of MR storytelling? (29:34)

I have different opinions on VR storytelling and MR storytelling.

I was at ILMxLAB developing a project on the life of Darth Vader (they’re going to release something next year, but I don’t know if it is the same I started working with)…let me put it this way: when you write a book, you create a fictional story. But the writer is a person, that has a life, that has had experiences, and often the story that he creates, even if it is fictional, is actually based on the real life of the writer. My theory is that VR is not necessarily the storytelling medium, VR is the experience medium, upon which someone may take things that happened to them, eventually process them, and affect the storytelling process maybe in another medium.

We can create two things: world logic, in VR or MR we can create some fantasies, create new worlds with their logic and you can also create situations: you come to this VR/MR place, and the world becomes aware of you… so I think that people will create situations for (other) people and the experience that you will have (in these situations in these worlds, with the world logic decided by the creators) will be itself the story.

At Oculus Studio they made an animated movie… but it is still on where you look and choreography on where you look… I think it’s elaborated and I think it’s better if the story is what your experience was. If you could look back and see what it has happened to you, that would be more interesting, especially if you are with other people, so every one of you can discuss about what’s happened to all of you… and I think that this is going to happen with mixed reality too. This is the story.

And if you’re a writer or creator, you may experience these simulations and this will affect how you will tell a story later on. And that’s as far as I got up to this point.

I think that people will run in circles trying to shackle VR and MR with cinema grammar because it doesn’t translate at all. We can appreciate cinematography and editing as languages onto themselves… if you put a child in a scene with blocks of ideas, the child would understand what is being communicated. One of the most powerful language that exists is editing and composition… it is not regarded as a language, but it is. You take this language out of VR and you are in this place.

Yeah, I know about gaze choreography and all the VR storytelling techniques used now, and that’s a way to do it… but I think that it’s more about the experience, YOUR experience and the product of that.

And what you are doing at Magic Leap to create these experiences for people? (36:30)

I’m not super focused on stories right now. I’m focused on form, I’m interested in form, like “hey, here is a new way to express yourself”. The form of the coming MR platform is moving, is evolving… but I don’t believe in a world where we are going to be in a room all the time… in VR everything is stuck in a room, and also MR at the moment… Pokemon Go was good because it let you move in the world, it was a great experiment, even if it was a simple one.

What I’ve been interested in my last year in Magic Leap is how we do move to city-scale MR experiences. What will we need to create such an infrastructure? Not by one company, not only by Magic Leap but by a lot of companies, that have different parts of the solution of registration of the virtual and real world… I mean, if I am at a street corner in Turin, and there may be 3, 4, 1000 applications put there by different groups, like the public, game universe, the City of Turin, etc…. and maybe the game universe has a layer running through this. These experiences may be of whatever sector: entertainment, transportation, health, arts and science, education… every sector can create a special application, and can be local to one spot or can span on a large destination, even the whole world.

The question is how are we going to register (the real and virtual world) and how we can verify the applications, the applications we trust and that we want to be part of, that will respect our privacy. Basically, it is like the internet moves to the world and that’s a big deal and will require a lot of people working together.

I’ve met a lot of storytellers because they know this is a new medium of expression, but they are only thinking about the story… while we should also think how to make characters persist in space. VR only gets truly interesting with social VR, where you interact and play together with people and so it feels alive. But it is problematic doing that in VR because it requires a lot of resources to do that all in CGI. So, storytelling becomes difficult if it’s not live.

So, this is the MagicVerse, that you are going to explain in your talk… (41:49)

Yes. But we’ll need lots of people to solve the problems. I’ll make questions because I don’t have the answers.

https://gfycat.com/AmusedDownrightBrownbear

An estimate of how much time do we need to get there? (42:11)

5 to 10 years. Doing some alpha-tests of larger scale deployment… 5G is coming on… 5G is going to be necessary, for immersive broadcasting with high fidelity graphics. Things will get serious in 2-5 years, with some 5G tests and that will be the beginning of a lot of interesting stuff. Then a lot of time will be needed to understand what works and what not, how people will use it, in which cities starting first.

Starting in 2 years, we’ll need 5-10 to become mainstream.