Mixed Reality: why it is cool and when it does fit your VR experience (…or not)
For this end of the year, I want to talk about one of the big trends we are experiencing in the immersive realities ecosystem: the growing attention to mixed reality. All the new headsets (starting from Quest Pro and Pico 4) are featuring color passthrough vision and this trend is here to stay because every headset that has been announced in the latest weeks has this functionality, too. All the major companies are inviting developers to experiment with augmented reality and it is also rumored that the upcoming Apple headset (which is coming next year, as usual) is based on this feature, too.
In this post I want to tell you my impressions about this feature, both as a developer and a user, hoping that it may be useful for you to understand the potential of mixed reality and if it is the case you employ it or not.
My experience with mixed reality
My team at New Technology Walkers launched its main game, HitMotion: Reloaded, in 2019 at the Vive Ecosystem Conference, in Shenzhen, China. I was there on the stage detailing how I managed to create augmented reality applications for the Vive Focus Plus, and at the end of the presentation, I launched the game.
In that period, I made lots of experiments with mixed reality on the Vive Focus: I did some tests on Aruco marker tracking so that to have AR elements over a marker; I applied some filters on the passthrough and with Enea LeFons created Beat Reality, which made your vision pulse at the rhythm of the music; I even made objects disappear with a very crappy implementation of what is called Diminished Reality. I can say I had lots of fun with it.
If you think about the timing, we are talking about 2 YEARS BEFORE the release of passthrough for developers on the Meta/Oculus Quest 2. At that time, we were just a bunch of developers that believed in this technology and that made hacked prototypes to experiment with their potential. Now things have become much easier, thanks to the implementation of passthrough in all most important SDKs, and also prettier, because of the integration of RGB colors in the visuals.
And of course, I haven’t stopped playing around with it: our fitness game HitMotion: Reloaded has been updated for Quest 2 and now recently also for Quest Pro, where it finally can shine with the best passthrough vision I’ve been able to experiment with. I’ve also made a little test regarding having a singer performing on my desk for VRROOM, and it was very interesting.
Should you implement mixed reality in your application?
Since I’m a big fan of mixed reality, you may think that this is a post about suggesting you implement it into your application. Actually, more than an enthusiast, I’m an engineer, and at the Polytechnic the teachers taught me to answer “it depends” to every possible question (this is usually good advice, but becomes a problem when my girlfriends ask me “Do you love me?”). So, should you implement mixed reality in your application? Well… it depends.
The fact is that MR is a technology, so a tool, and as is, should always be implemented where it is really needed. The question so becomes if MR is needed by your application or not. This doesn’t change the fact that “it depends”.
Mixed reality has a “different language” than virtual reality. It is a technology that blends your real environment with virtual components, which gives you totally different sensations from virtual reality, that instead puts you entirely in another environment. If you are making a product, it may fit one of the two or maybe even both depending on if your experience fits both of these situations. For instance, if it is important for you to create the right atmosphere (like in a horror game) or to make the user feel somewhere else, then virtual reality is what you are looking for. If you have a local multiplayer game, instead, making you see the other players with passthrough could be cool. Let me tell you about some of the experiences I had to give you some examples of what I mean by “could fit or not”.
HitMotion: Reloaded
HitMotion: Reloaded has born as a mixed reality game for the Vive Focus Plus. We chose mixed reality for two reasons: the first one is that we were experimenting a lot with it, and we thought it could be cool to launch a game with it to differentiate ourselves from the other games launching on Focus Plus. The second reason was safety: in our game, you have to punch virtual objects fast, and the risk is that you punch all the people around you. With mixed reality, you see what happens around you, so you don’t risk punching anyone… unless this is exactly what you want to do (and even in this case, mixed reality is still better because you can aim better at their faces).
All of that made sense, and me and Max (my cofounder) understood that we took the right decision when we went to Nanchang to showcase our game. No people had accidents while playing our games… plus everyone was super-surprised by the fact they could see through the headset, and we made a great impression, even if the passthrough was so-so.
I would say anyway that the advantages of using mixed reality stop more or less here. Seeing the target I have to punch in my living room doesn’t add much to the experience, because there is no real blending between the two. The targets are in my real place, but I totally ignore that and I just think about punching them. We tried to create some connection by making the real environment change color depending on the punch result (e.g. green passthrough edges to signal a very good punch), but with the fast pace of the game, I don’t give much attention to that, either.
Playing in the two modes while doing the development gave me anyway some interesting insights into the difference between the two modes. In Virtual Reality, I feel more unsafe while playing the game, but at the same time I like that there is a full context for the game. Our artists spent some time recreating this “street environment” where playing the game, and I think the setting is quite cool. Virtual Reality helps in giving more context about why I’m there punching stuff, it helps me in feeling there, in feeling the experience as more “complete”. I’m in this rusty ring, and I punch stuff together with my robo-trainer. This is cool.
Augmented reality makes the game more “essential”. I have only the targets I have to punch. I lose the lore completely, I just see the bare minimum I need. On the practical side, it is much better to play it in mixed reality, because I don’t disconnect from my reality and I can play it safely.
In the two modes, it is so two different versions of the same game. And they could even evolve in different ways: for instance, in AR it could be possible to make people train in front of a real boxing bag, with some augmented reality targets showing over it. In VR, it could evolve more by making the environment react properly to the game progress. In the long run, the two different modes could even become two different games.
I have also to add a last bit of info about my experience with HitMotion. Recently I got the Quest Pro and I modified the game so that to make it compatible with its passthrough. Trying the game with passthrough color was like night and day for me for what concerns the mixing of real and virtual elements. You would maybe be surprised by the fact that this enlightenment didn’t come from the target I had to punch. Having a target in my living room, considering also that its life is like 0.8 seconds from when I see it to when I punch it, wasn’t that interesting. What really surprised me were the gloves and Bob, the robo-trainer.
The first time I tried the color passthrough, and I could see the virtual gloves superimposed on my real hands, I was surprised by how much my brain started to identify them as mine. I have never had a similar sensation with AR glasses, because they are semi-transparent and with a low FOV. I never had it with Quest 2 either, because the black and white of the passthrough vision clearly separated real elements from virtual ones. But with color passthrough, seeing the yellow and cyan gloves on my hands, made my brain feel a bit like they were really on me. I had to move a chair that was in my play area, and my brain thought for a split second “but can you move the chair if you are wearing boxing gloves?”. It was kinda mindblowing. I’m not saying that it is fully realistic (it is not), I’m saying that it started being realistic enough to start tricking my brain and this felt already cool. Mixed reality helped me in immersing more into the game because it made me feel more like a boxing athlete. I so understood the potential of playing with the “self” in mixed reality. I saw yesterday Mike VRO sharing a video on Twitter about someone wearing a rifle in mixed reality… well, I think that could be another cool thing to do on the Quest Pro, because you would feel more like holding that weapon for real in an action game.
The other thing that I liked was when Bob appeared: seeing this nice robot in my bedroom, there, at 1m from me, teaching me boxing was another cool sensation. He felt more real because I felt him like being at home with me. He was out-of-context: a robot fits better in a steel environment than a bedroom, but he felt more connected with me because he was not in some kind of abstract non-existent street, but he was there with me in my real room. It is like the real room transferred some of its “realness” to the virtual element.
Playing HitMotion with Quest Pro made me realize that apart from the environment, color mixed reality can help in creating a better connection between the virtual elements and the player and this can give amazing sensations.
(… and if you want to feel them, please download our game from App Lab and give us a good review!)
Beat Saber
When Zuck announced the Quest Pro, he showcased during the Meta Connect that Beat Games was experimenting with a mixed reality version of Beat Saber. He then showed a short video of it, and it didn’t look that great.
The reason is simple: Beat Saber is what it is because all the visual polish it has of the environment around it. It is just beautiful. If you remove the environment from the equation, the game becomes less cool, in my opinion… unless you substitute it with something else. Considering the experience I had with HitMotion, I think that a MR Beat Saber could be great for the sensation of really feeling lightsabers in your hands. We all dreamt of owning a lightsaber after having seen Star Wars, and with this game, it could finally be possible. So if Beat Games could play more around this concept, it could make the game more interesting. Or another approach could be substituting the visual VR effects with some effects that happen in your room… provided that your game could have some understanding of your room’s shape.
In any case, a mixed reality version of Beat Saber would require some effort: you can’t just take a VR game, remove the background and call it a mixed reality game.
Demeo
Resolution Games has just launched an MR mode of Demeo, and many people claimed that Meta should have showcased this instead of Beat Saber at Meta Connect. I totally agree with them. Demeo is basically a VR board game, and people usually play board games on their table, with all their friends around. If Demeo becomes in mixed reality, you can play it on your table, seeing all your friends that are playing it together with you for real around you.
In this case, the good thing about passthrough is that it helps in giving more the sensation that you are playing that game “like in real life”. And especially that playing it you can see all your friends around you, and so being able to use natural interactions with them, like appreciating all the facial expressions that can’t be re-created in VR, yet.
Spatial Ops
Talking about Resolution Games, they recently announced Spatial Ops, which is basically a local multiplayer shooter: 4-6 people can stay in a large space that gets filled by virtual elements and start shooting each other, laser-tag style. This game has sense in mixed reality for two of the reasons described above. One is safety… thanks to MR, you can see the space around you, so you don’t risk bumping on other people, or on objects in the space. And the other one is the ability to see your friends for real, and so have a better social connection.
Sex Like Real
Ok, so it’s time to address an important topic in mixed reality. Whenever I talk about immersive reality technologies, my non-techie friends always say “imagine the porn about that!”. The reality is that usually, I don’t have to imagine them, because I already saw them. Or better, sorry… A FRIEND OF MINE saw them and told me they are cool. And even with mixed reality, there is no exception to this rule, this prolific friend of mine, spent 1-2 hours inside Sex Like Real (inside DeoVR) to be able to talk about them to the community. Sometimes being a technologist is hard. Pun intended.
SLR has recently added a passthrough mode, in which basically you can select one of their 180 VR videos and apply some sort of chroma-keying to segment the parts of the scene you are most interested in. This is the first foray into adult MR entertainment and should be enough to be a taste of what it could be in full AR mode.
The current implementation is pretty rough, especially because most of the videos have not been shot with this in mind. This means that for instance, the background of some of the suggested passthrough videos is black… but the performer has black hair, brown eyes, and black clothes. The result is that when you try to apply chroma keying of the black color, you see in front of you a girl with no eyes and when she opens her mouth you see a hole in her face that lets you see the walls behind her. More than getting aroused, you get PTSD from being locked in your home with a horny horror creature. Or other times the girl is cut-out in an ok-ish way, but you still see some objects of the setting of the scene: in a kitchen scene, for instance, all the walls were white, so I could remove them easily, but I still saw floating around me a bowl of pears. And why look at a girl when you can watch a bowl of pears?
There are anyway some videos in which proper segmentation is already possible, and also recently the company started shooting some with green screen specifically for this purpose. When watching them, I had a reaction similar to the one I had while playing HitMotion. The virtual reality version of the video is good because gives you context. Oh, I’m a pizza delivery and she has not the money to pay. Ok, nice (but couldn’t she just use a card to pay? Whatever…). But the augmented reality version is interesting because it gives more sense of connection with the performer. It is less “I’m in this scenario doing this rolepay” but it is more “there is this person I invited at home”. It’s more intimate, more like she (or he) is there with you to have an intimate moment. Even when she approaches the camera, it is more like she is interacting with you for real.
It’s a different sensation. It was interesting reading in the forums of the app many “men of culture” discussing how they were using passthrough. I remember one had even found the right video so that while lying in the bed, he could have the impression with passthrough that the actress was cuddling in the bed with him. Which was probably the same thing he could see in VR, but in AR felt more real, more intimate, more as if this was happening in real life, because he was in his real bed. Again, the real element giving part of its “realness” to the virtual element. I noticed that many were looking for videos that created a sense of connection with the performer, more than super-kinky stuff. And it makes sense, because again: with VR you could have this journey where you are in a certain place and something happens, but AR is the one that can give more the impression that something is happening in front of you in real life.
So again, even in this field, the sense of self-connection or the connection with the place you are in are relevant for mixed reality.
Environment interactions
There are in the end some apps that use passthrough for utility reasons, to let you interact with objects in your environment. For instance, Horizon Workrooms uses passthrough to let you see a part of your real environment, so that you can see your keyboard and use it inside the virtual application.
I guess the idea of using real objects to interact with VR can become even better as soon as there will be environment understanding and object detection. If the system can understand the environment around you, you can interact with your environment to interact with the MR app. And that could be very cool.
In the meanwhile, you can put a bit of effort yourself to simulate it. Shopify recently made this demo showing how to assemble a piece of furniture in the Quest Pro. If you use the real elements where there are the virtual instructions, you can have the system help you in assembling furniture in a very handy way. The utility of AR/MR is going to skyrocket with similar applications in the future.
Clippy
Talking about experiments with MR, why not see Clippy in your real place? Ok, just kidding of course.
Do your choice
What I wanted to highlight in this post is the different nature of applications implementing mixed reality. It is a bit how we discuss games made for VR versus VR portings: you notice a game that has been designed with VR in mind because it “speaks the VR language”, it has all the interactions made for VR, it has the right framerate, it doesn’t give you motion sickness and such. And the same hold for AR/MR: I think that this technology has a different language than pure VR. The two technologies have some overlaps, but they are not the same. So when thinking about the MR support for your app, think about if it makes sense or not, if you can convert it to that language.
In this post, I highlighted a few typical features of MR apps, but I’m sure there are more I didn’t even think about. My suggestion is to trust your instinct: if in doubt, make a crap prototype of an AR version of your app, and let all your team try it. If you all see a compelling reason to have it, then go all for it. If it feels just like a gimmick, a “let’s do it because it’s the thing of the moment”, then avoid doing that and maybe do your next app all focused from scratch on MR.
That’s my suggestion. But feel free also to tell me your impressions: my comment section here below is waiting just for you!
(Header image by Meta)
Disclaimer: this blog contains advertisement and affiliate links to sustain itself. If you click on an affiliate link, I'll be very happy because I'll earn a small commission on your purchase. You can find my boring full disclosure here.
