john gaeta magic leap interview mixed reality magicverse

Magic Leap’s John Gaeta talks about the future of cinema, storytelling and mixed reality

One of the best moments ever at View Conference 2018, has been when I was able to interview John Gaeta. If you don’t know who John Gaeta is… well, he is the inventor of the bullet-time of the movie The Matrix.

Now, this genius that has revolutionized cinema has landed in the mixed reality ecosystem, to try to improve it as well. He has joined Rony Abovitz at Magic Leap and now he’s there, trying to build the MagicVerse as the SVP of Creative Strategy of the Florida company.

I’ve had the opportunity to interview him together with two other journalists and during the interview we have been able to span different topics, from cinema to the MagicVerse, not to mention mixed reality storytelling and immersive cinema. While being there next to him, I could really feel the power of his knowledge and my mind has felt enriched by the discussion with him.

So, why don’t you participate in this amazing discussion with me? Thanks to the help of my friend and colleague Leandro, I’ve recorded all the interview so that you can listen to the exact words told by John Gaeta and feel as if you were there at just some centimeters from him.

As I did with Mister President Alvin Wang Graylin, I’ll leave you here the full video of the interview, with a non-exact (simplified) transcription of it below it, so that you can choose how to get to know Gaeta’s thoughts. You can also skim through the written questions and click on the link to enjoy the video version of only the questions you are most interested in. And if you are super-lazy, you can directly go to the end of the article and just read the key points of the interview.
Before going on, I’m sorry if in the transcriptions may be some errors… sometimes I’ve still difficulties in understanding spoken English when the audio is not perfect.

That said… enjoy the mind-opening John Gaeta’s words!

I would like to ask you when and where you started working in the movie scene. Is it true that you started working on Saturday Night Live? (0:07)

Yes, it is true: it was a spectacular job: I was involved in all the short movies that they made. This introduced me to lots of interesting people and taught me a lot about fast writing and production of short comic movies.

At the same time I was living in Greenwich Village and I had some talented friends working in the advertising sector, making things like stop-motion animation for advertising… and these friends told me that a new studio was being built in Massachusetts by Douglas Trumbull. Douglas went away from Hollywood to create this super-studio with all the cutting-edge technology (like robotic camera systems, computers, etc…). All sort of innovations were happening in that studio, so my group of friends pulled me there and I got a job in the camera department. Within a year, I was already shooting scenes with a running camera.

Douglas was not only focused on movies but on immersive media, in creating the future of cinema and he invented a lot in this sense (like the Imax format). He was obsessed with Virtual Reality.

john gaeta magic leap interview mixed reality magicverse
Innovative cinematographic hardware used by John Gaeta

He was like a mentor for me, he taught that if something doesn’t exist, you can make it yourself.
When he took the Imax route, he left us in his studio, but without him anymore, and we asked ourselves what to do. We did two movies: one was What Dreams May Come, that has won an Academy Award, and the other one was The Matrix. And in three-four months, our small team made both the painting effect of What Dreams May Come and the bullet-time effect of The Matrix.

For Robin Williams, there was a fantastic look, fantastic special effects… (6:34)

Wonderful. One of the colleagues of mine of that time was Kim Libreri, that is now the CTO of Epic Games and he created the optical flow for the paint effect for What Dreams May Come.

For the Matrix trilogy it was important the use of computer vision, immersive photography, volumetric capture, the first markerless motion capture, to create frame by frame models of the performances… and today all these techniques are prolific in virtual reality and mixed reality. And all these things connected all different kind of people there, and we worked together with different skills on the same things.

After The Matrix, you made Speed Racer. Can you talk about that movie? (8:23)

It was one of my favorite projects. We thought that it was more important to work on how the movie could make you feel than in making it seem real, so everything was developed so to be artistic and expressive. That movie has an interesting combination of 3D, 2.5D and 2D mixed together. We went to Europe and found some amazing locations for the movie.

I am more an idea person, I like to work with talented engineers and artists… but you have to create such an amazing team by hand-picking them. The team has to be small and be able to work both on the creative and technical side. With Speed Racer, the team was a very high-level visualization team.

How has Matrix changed the perception of the audience at the cinema? After that movie, do all people want always to see that kind of cool special effects? (11:58)
john gaeta magic leap interview mixed reality magicverse
The Matrix is a movie that has inspired an entire generation of people, me included

I have to say, I don’t care really much about special effects, I’m not interested in special-effect movies, unless they’re interesting, unless they tell interesting stories. Matrix resonated for a couple of reasons:

  1. It came out in the 90s, and it was the period that the average people were accepting the Internet. It took time for people to accept the Internet and not fear it anymore. The movie came when people started thinking about a digital world where we are together (the Internet), where we were going beyond the inflection point of mainstream adoption of the Internet. The science fiction leap was “You are inside the internet now. The virtual reality means being in the Internet”. Such revolutions do not happen often, and now this is happening again, with VR, AR… but regarding immersive technologies, apart from maybe Pokemon Go, we have not seen something mainstream up to now, we are not at the inflection point.
    In the last 5 years, I’ve seen another movie having such an impact, but not in the mainstream audience, but in the technology sector: “Her” by Spike Jonze. He figured out that Artificial Intelligence would have been humanized at some point, in a period when people didn’t even know what AI was. It resonated because techies were experimenting with AI interfaces, and the movie made these people understand that someone may also fall in love with these AIs. That rippled in many Silicon Valley labs, and made people think “Uhm, how can I get there?”.

  2. The early bullet-time effect was something a bit like quick-time-VR, but a much higher level… you know, it is impossible to cheat time and space with a camera. People knew it was not possible to do that: the film resonated because only in a simulation, only in virtual reality you can cheat time and space. In 5 years or such, in VR we will be able to do that every time we want.

At Lucasfilm we made an incredible high-resolution high-fidelity very-close-to-reality looking tests… and you may have seen Mica, the AI human that Magic Leap put out… this is the best virtual human that I’ve seen and it is actually real-time.

Matrix resonated because the directors were openminded and allowed us to use the techniques that we wanted. They were ok in letting us experimenting and taking risks. Today the movie industry is a zero-risk one. And so we tried techniques that we thought were ideal for actual virtual reality, we captured the world both from inward and from outward. Thanks to hard work and luck we managed to do a great job in that. It is possible for people to create these innovative things in AR, VR, movies if they are allowed to innovate.

Why do people need VR? (20:38)
virtual reality selfie vive focus
To need or not to need. This is the question

I think people will want and need Mixed Reality. Virtual Reality can be transcendent and transformational for people because it disconnects you and it can change people… but I’m not sure we need it. It’s definitely a choice.

But Mixed Reality… there are amazing things that can be made… but as every medium it has good and bad sides… even radio, TV (it could be great for education… oh no, it’s a propaganda tool) or Internet (place of knowledge, and in the end it is full of hate and distortion). VR can take you to a spectacular place you will never be able to go, but Mixed Reality can be prolific, it has the potential to go mainstream, it has the ability to make people come together, something that in VR is difficult to do, because VR encloses you in a headset and so you don’t see people surrounding you and the place around you… but it is a great expression and artistic medium, maybe like no one other ever. Mixed reality preserve the opportunity to gather together, collaborate together, educate, in a very powerful way, also because the application can have a powerful backend in the cloud, with remarkable processing power. But the key is people being together. VR is not, but mixed reality is going to be the next medium beyond television, that will be everywhere.

And will we see some kind of VR viewer inside the cinemas in the future? (24:27)
Yue Cheng VR Cinema Beijing
VR Cinema in Beijing (Image by Yue Cheng Media)

I worked on that for years, it was one of my obsessions. We make movies and movies are very technically driven, and as real-time computer graphics will start having a quality comparable to the one of slow-rendering one, we’ll start seeing lots of content produced inside engines, game engines. The world “game engine” has to be changed in “real-time graphics engines” and these will be the place where we will have volumetric photography, photorealistic scenes, etc… All the scenes will be produced in those engines and so the scenes will be of course exported for the cinema, but you can offer people the opportunity to pick a scene and let them teleport inside it so that they are inside the scene… I’ve done tests on this for years at Lucasfilms … you can return back to a particular scene and enter there and move inside there. Movies can have portals.

Is this something that ILMxLAB is working on? (26:47)

Yes, let’s see how they will do… but also others are working on it.

Once you have the cinema rendered in a real-time engine, you can do a lot of things with it: you can enter into scenes, you can project it holographically, as like in an immersive CAVE, or using glasses. You can have this immersive experience where you see the spaceships of Star Wars dropping droids around you… and this could use exactly the same assets from the film: it is not a copy, a remake… it is not the movie made by a company and the XR experience made by an external game studio that has not participated in the making of the movie. It’s the same crew that can do both.

john gaeta magic leap interview mixed reality magicverse
Experiments on immersing people inside the scenes of movies
We talked about VR storytelling and there are cool experiments with it, but I have not seen yet compelling storytelling content in AR and MR. Since you are inside Magic Leap… what are you doing to make compelling mixed reality content and what is the future of MR storytelling? (29:34)

I have different opinions on VR storytelling and MR storytelling.

I was at ILMxLAB developing a project on the life of Darth Vader (they’re going to release something next year, but I don’t know if it is the same I started working with)…let me put it this way: when you write a book, you create a fictional story. But the writer is a person, that has a life, that has had experiences, and often the story that he creates, even if it is fictional, is actually based on the real life of the writer. My theory is that VR is not necessarily the storytelling medium, VR is the experience medium, upon which someone may take things that happened to them, eventually process them, and affect the storytelling process maybe in another medium.

john gaeta magic leap interview mixed reality magicverse
John Gaeta worked at a Darth Vader experience at ILMxLAB where the purpose was making people explore the castle of Darth Vader

We can create two things: world logic, in VR or MR we can create some fantasies, create new worlds with their logic and you can also create situations: you come to this VR/MR place, and the world becomes aware of you… so I think that people will create situations for (other) people and the experience that you will have (in these situations in these worlds, with the world logic decided by the creators) will be itself the story.

At Oculus Studio they made an animated movie… but it is still on where you look and choreography on where you look… I think it’s elaborated and I think it’s better if the story is what your experience was. If you could look back and see what it has happened to you, that would be more interesting, especially if you are with other people, so every one of you can discuss about what’s happened to all of you… and I think that this is going to happen with mixed reality too. This is the story.

And if you’re a writer or creator, you may experience these simulations and this will affect how you will tell a story later on. And that’s as far as I got up to this point.

john gaeta magic leap interview mixed reality magicverse
According to John Gaeta, Carne Y Arena is an amazing VR experience

I think that people will run in circles trying to shackle VR and MR with cinema grammar because it doesn’t translate at all. We can appreciate cinematography and editing as languages onto themselves… if you put a child in a scene with blocks of ideas, the child would understand what is being communicated. One of the most powerful language that exists is editing and composition… it is not regarded as a language, but it is. You take this language out of VR and you are in this place.

Yeah, I know about gaze choreography and all the VR storytelling techniques used now, and that’s a way to do it… but I think that it’s more about the experience, YOUR experience and the product of that.

And what you are doing at Magic Leap to create these experiences for people? (36:30)
john gaeta magic leap interview mixed reality magicverse
Magic Leap glasses. During the talk, John said that he and Rony Abovitz were already penpals. He remained astonished when he went to Magic Leap HQ and trying the big Magic Leap prototype saw that he was able to focus his eyes on a virtual shark

I’m not super focused on stories right now. I’m focused on form, I’m interested in form, like “hey, here is a new way to express yourself”. The form of the coming MR platform is moving, is evolving… but I don’t believe in a world where we are going to be in a room all the time… in VR everything is stuck in a room, and also MR at the moment… Pokemon Go was good because it let you move in the world, it was a great experiment, even if it was a simple one.

What I’ve been interested in my last year in Magic Leap is how we do move to city-scale MR experiences. What will we need to create such an infrastructure? Not by one company, not only by Magic Leap but by a lot of companies, that have different parts of the solution of registration of the virtual and real world… I mean, if I am at a street corner in Turin, and there may be 3, 4, 1000 applications put there by different groups, like the public, game universe, the City of Turin, etc…. and maybe the game universe has a layer running through this. These experiences may be of whatever sector: entertainment, transportation, health, arts and science, education… every sector can create a special application, and can be local to one spot or can span on a large destination, even the whole world.

john gaeta magic leap interview mixed reality magicverse
The definition of the MagicVerse: it is a mixed reality world-scale metaverse using different technologies
john gaeta magic leap interview mixed reality magicverse
As you can see, in the MagicVerse, in every moment, in every place, there will be a lot of mixed reality applications running. We can see them as running at different layers, divided by their sector of reference

The question is how are we going to register (the real and virtual world) and how we can verify the applications, the applications we trust and that we want to be part of, that will respect our privacy. Basically, it is like the internet moves to the world and that’s a big deal and will require a lot of people working together.

I’ve met a lot of storytellers because they know this is a new medium of expression, but they are only thinking about the story… while we should also think how to make characters persist in space. VR only gets truly interesting with social VR, where you interact and play together with people and so it feels alive. But it is problematic doing that in VR because it requires a lot of resources to do that all in CGI. So, storytelling becomes difficult if it’s not live.

john gaeta magic leap interview mixed reality magicverse
Attributes of the MagicVerse
So, this is the MagicVerse, that you are going to explain in your talk… (41:49)

Yes. But we’ll need lots of people to solve the problems. I’ll make questions because I don’t have the answers.

john gaeta magic leap interview mixed reality magicverse
Definition of Magicverse
An estimate of how much time do we need to get there? (42:11)

5 to 10 years. Doing some alpha-tests of larger scale deployment… 5G is coming on… 5G is going to be necessary, for immersive broadcasting with high fidelity graphics. Things will get serious in 2-5 years, with some 5G tests and that will be the beginning of a lot of interesting stuff. Then a lot of time will be needed to understand what works and what not, how people will use it, in which cities starting first.

john gaeta magic leap interview mixed reality magicverse
5G will be fundamental for the development of world-scale XR technologies

Starting in 2 years, we’ll need 5-10 to become mainstream.


And that’s it with the interview.

Some key points of Gaeta’s interview that I want to highlight:

  • To do extraordinary things and change the world, you need a very focused high-quality team and especially you must be allowed to take risks. Without open minded leaders, we’ll stick in just repeating what we already do, in doing standard but profitable things. Gaeta detailed in the talk that he abandoned the movie sector because no one wants to take risks, they just want to make blockbuster movies to make sure money;
  • Cinema has introduced some techniques that are now being used in virtual reality. And in general, the world of cinema and of “gaming” will converge always more and in the end, we’ll have movies that will be realized with game-engines like Unity and Unreal Engine. This will make possible for the viewer to enter inside the scenes, to be inside the movies and move inside it. It will be amazing;
  • Game Engines should be called “real-time graphics engines” because now they serve a lot of purposes and not only games;
  • VR is a great artistic tool. But VR encloses people and makes them stay in virtual rooms. MR is better because it makes people collaborate together with the people surrounding them. MR also allow people to move in the world, to have world-scale immersive systems.  MR is the technology that will become mainstream;
  • People are trying to transpose cinema to XR, but this is damn wrong. XR has a different grammar from the one of cinema and we shouldn’t try to take cinema to XR, but to make people live storytelling experiences in a completely new and different way in XR. Gaeta thinks that the key is creating virtual worlds, virtual experiences with virtual characters and then just make the user live his/her own adventure inside there. The personal experience that the user will live there will be the story, will be his/her own story and if this can be shared with other people, this is even better;
  • With Mixed Reality itwill be possible to create something new that is the MagicVerse. The MagicVerse is basically our world, augmented by different AR experiences that are persistent, shared with other people and that have a physical mapping between the virtual and the real. If you are familiar with the term AR Cloud… well, it is something like that. In the MagicVerse there will be applications of all kind, from entertainment, to healthcare, not to mention transportation, art, etc…

    john gaeta magic leap interview mixed reality magicverse
    The real world and the virtual world will be deeply interconnected, and at every moment we will operate both as a real person and as a digital avatar
  • The MagicVerse can’t be built by Magic Leap alone: it requires a lot of expertises from a lot of different people and companies that will have to cooperate to create it. It will be an open ecosystem;
  • 5G will be fundamental for building the MagicVerse;
  • We’ll need 5-10 years to get to such a mixed reality world.

So much food for thoughts.

john gaeta magic leap interview mixed reality magicverse
Magic Leap 4 “North Stars”, that is the basic concepts onto which they build their vision

I really want to thank a lot John Gaeta for the time he has spent with us, explaining to us all these important concepts. I really hope to be able to meet him again in the future, to have some updates on the progress he has made in developing the mixed reality world.

I hope that you liked this interview and that you found it interesting as I did. And don’t forget to tell me your opinions about his thoughts here in the comments or on my social media channels!

(And also don’t forget to subscribe to my newsletter to receive such cool interviews whenever I’ll publish them 😉 )


Disclaimer: this blog contains advertisement and affiliate links to sustain itself. If you click on an affiliate link, I'll be very happy because I'll earn a small commission on your purchase. You can find my boring full disclosure here.

Releated

playstation vr 2

The XR Week Peek (2024.03.26): Sony halts production of PSVR 2, Meta slashes the price of Quest 2, and more!

This has been the week of GDC and of the NVIDIA GTC, so we have a bit more pieces of news than the usual because there have been some interesting things announced there. Oh yes, also IEEE VR had its show. Actually, a lot of things happened for tech people in the US, which is […]

We need camera access to unleash the full potential of Mixed Reality

These days I’m carrying on some experiments with XR and other technologies. I had some wonderful ideas of Mixed Reality applications I would like to prototype, but most of them are impossible to do in this moment because of a decision that almost all VR/MR headset manufacturers have taken: preventing developers from accessing camera data. […]