The 300th XR Week Peek (2023.04.10): Meta innovates image segmentation, Chrome implements WebGPU, and more!
It’s the 300th edition of my XR Week Peek. A lot of time has passed by, and I can’t even recall all the countless days I’ve spent reading news about XR and then summarizing them at the end of the week. It’s been a lot of effort, that required me a lot of discipline every week: every month it becomes more difficult to keep this column alive because my workload is increasing, but until now I managed to pressure myself to fulfill my duty every week.
I’m not going to celebrate today, but I feel very thankful for what happened. I want to thank all of you that are reading this column every week and that gave me a reason to continue to carry on this big work over all these years. I thank you from the bottom of my heart. I hope that we can arrive together to the 400th episode in two years… let’s see 😉
If you want to give me a tip for all these hard years at work, you can join my Patreon. Otherwise, feel just free to reach out to me and send me good vibes, I appreciate the same 🙂
Top news of the week
Meta announces Segment Anything
Meta has announced “Segment Anything”, a computer vision system driven by AI that can segment whatever object in a scene. This means that given whatever image and clicking on whatever pixel of this image, the system is able to identify what is the area of the “object” that pixel belongs to. So if you click on the door of a house, the system can identify the whole door and separate it from the rest of the image. This research work is disruptive because it can detect whatever kind of objects, even the ones it has never seen during training. It has learned the “general concept of what an object is” and so can isolate every object from the rest of the image.
There is an online demo for this system, and you can try it out either with preset or custom images. I’ve tried it out myself, and it works kinda well.
You may wonder why this system is relevant for us XR people: well, Meta itself talks about AR applications for this system in the launch post. AR glasses may separate the various objects that compose an image they see with their cameras and so be able to detect what the user has around him/her. Glasses could also highlight the objects the user has in front of him/her during a training experience or while he/she is following a certain recipe. This research, taken further, could bring many interesting innovations in the AR field.
More info (Twitter post that describes the excitement around this project)
More info (Official launch blog post)
Other relevant news
Chrome ships WebGPU
Chrome, the most popular browser around, has just implemented WebGPU. As the Chrome team explains “WebGPU is a new API for the web, which exposes modern hardware capabilities and allows rendering and computation operations on a GPU, similar to Direct3D 12, Metal, and Vulkan. Unlike the WebGL family of APIs, WebGPU offers access to more advanced GPU features and provides first-class support for general computations on the GPU”. Translated: WebGPU allows for better integration with GPU features, offering more capabilities to 3D apps running on the web.
I see this as a positive evolution that can favor also WebXR. We all see the potentialities of WebXR, in particular because it can reduce the friction of use of XR (no need to install apps, you just open a link)… but until now WebXR applications have always shown sub-par graphics if compared to standalone ones. I guess that WebGPU can help in improving this in the long run.
WebGPU has just been shipped with Chrome 113, and Google has promised that it will keep building on this foundation to increase and improve this feature in the upcoming months. Let’s see if this will bring real benefits to WebXR.
Mojo Vision raises $22M to build the displays of the future
Mojo Vision, the company that became famous for its goal of building smart contact lenses, has just announced that it received a new funding round of $22M to pursue its new goal of building the displays of the future.
A few months ago Mojo had to scrap its plans about contact lenses to pivot to build micro-LED displays. When building the smart lens, they had to design a very small display, able to pack 14,000 pixels per inch (ppi). From this foundation, the company plans to build very small micro-LED displays with very high pixel density which may be used in many fields, including AR and VR.
Mojo has just restructured its internal team to focus on displays. It has developed a proprietary High-Performance Quantum Dot (HPQD) technology to make very small, very bright, very efficient RGB pixels.
The company’s technology includes the following advances (let me do some copy-paste):
- Dynamic displays up to 28,000 pixels per inch
- Efficient blue micro-LED devices at sub-µm scale
- High-efficiency quantum dot ink for red and green
- High brightness at 1M+ nits
- A display system that incorporates an optimized CMOS backplane, wafer-to-wafer bonding, and custom micro-lens optics
- A high-volume manufacturing process that is based on 300mm gallium nitride (GaN) on Silicon, and an end-to-end 300mm flow
Even if it is not shooting at the moon anymore, I still hope this company can be relevant for our XR field. Good luck, Mojo Vision!
More info (Mojo Vision gets $22M to build micro-LED displays)
More info (Mojo Vision’s official press release)
7000 people connected together to Yuga Labs’ Otherside
Yuga Labs’ “Otherside” web3-based world just had another massive test run demo for its users. And, even if the experience per se was quite simple, it went very well.
More than 7,000 concurrent users could join the test and be in the same virtual space together. This is impressive: remember that VRChat supports up to 40 people in the same room (technically up to 80), and this experience managed to arrive at 7,000. This kind of networking stack is what we need to make the “metaverse” possible, to have full virtual cities inhabited by people that can all see each other. Sure, this experience is not VR (so some data could be simplified), but it’s an impressive result nonetheless. We need more of this in the future.
News worth a mention
Let’s have realistic expectations on the Apple headset
We are all very excited about Apple joining our industry, but we must have realistic expectations about the headset that is going to come. Karl Guttag, the display expert, has just written an article reminding us that despite its money and innovation power, Apple can’t defy the laws of physics, and so can’t solve some problems of XR about which there is not an available technology yet. Mixed News reported some numbers of the current sales of VR headsets, with Quest Pro shipping probably 300K units and Pico selling 40% less than expected. And Apple Reality Pro, expected to come at $3000 or more, is improbable that is going to change this trend.
So let’s not hype this headset as the savior, and let’s give it a few iterations to become relevant, as it has happened with Apple Watch.
More info (Apple can’t defy the laws of physics)
More info (Current disappointing sales of headsets)
The Metaverse Fashion Week has taken place
The second edition of the Metaverse Fashion Week has just finished. The event about digital fashion and web3 has taken place in Decentraland, Spatial.io, and Over the Reality. It was an event probably more interesting than last year, and that spanned across multiple digital worlds. It involved many famous brands: Tommy Hilfiger, Ben Bridge Jeweler, and Adidas were the three reporting the biggest success. But the overall event had far fewer visits than last year, most probably because the metaverse hype has deflated.
More info (Metaverse Fashion Week)
More info (Tommy Hilfiger releases its cross-metaverse hub)
More info (Brands’ performances in the Metaverse Fashion Week)
Lifecast makes VR videos volumetric
I discovered on Medium that Lifecast has an interesting technology that takes a standard 180 VR video and makes it volumetric. That is, it transforms a 3DOF video into a 6DOF one into which you can move your head. The basic idea is to separate the elements of the video into 3 “layers” using a depth map, and then reconstruct the missing pixels occluded by each layer on the next one using an AI technology like Nerf. This way, you can move your head, and enjoy the parallax between the three layers, for a decent 6DOF effect. It seems like an interesting technology, and I would love to try it.
Owlchemy Labs thinks hand tracking can make VR mainstream
Owlchemy Labs has just presented a demo of the popular game Job Simulator completely working through hands tracking. The company has fine-tuned the interactions in the best way possible so the game can be played without controllers by retaining a good experience. Owlchemy is convinced that hand tracking is what will pave the way to make VR mainstream, because it makes interactions more natural, intuitive, and frees us from controllers.
Personally, I think that hand tracking is amazing for some kinds of interactions, like UI interactions, but is usually not suitable for other use cases, like playing FPS games. Plus, VR has so many other problems that prevent its mainstream adoption, that hand tracking doesn’t seem to me to change its situation much. But still, the work that Owlchemy is doing is very important on the UX side, so I appreciate it.
John Carmack talks with Boz about VR
Andrew “Boz” Bosworth has started its new series of podcasts with an interview with the god John Carmack. John has talked about AI, VR, and the future of technology in general, and I’ve heard that the whole podcast (1h long) is worth to listen. One of the interesting tidbits of it is that Carmack believes that one of the big problems of VR is friction: the time needed to turn on a headset, find the controllers, set up the guardian, run a game, connect, and all the rest is too long for the technology to be mainstream. I totally agree with him (of course… can Carmack be wrong? Impossible!)
NuEyes announces XR glasses for dentists
NuEyes has just announced NuLoupes, which is a new pair of augmented reality glasses for dentists. These glasses can provide dynamic zoom 1–10x of what the dentist has in front of his eyes, plus perform object recognition and let the dentist write/read textual notes without using his hands. It seems a pretty interesting use case for XR.
More info (NuEyes NuLoupes) More info (Official website)
The Upload Showcase is back!
The Upload Showcase (Summer Edition) is coming back on June this year. That day, Upload will announce many interesting news about VR games, trying to make some buzz about virtual reality in general. The popular magazine is accepting pitches of news to talk about, so if you are making a VR game and want to promote it, try contacting them.
Some news on content
- Gorn is now available for PSVR 2. According to its publisher, the game has sold more than 1.5 million copies across all platforms
- Fruit Ninja VR 2 has been released on April 6 for Quest, Pico, and PC VR
- Creed: Rise to Glory Championship Edition is available now for PSVR 2 and Quest 2. It includes characters and locations from the new Creed III movie
- Road To VR reviewed Vertigo 2 giving it a whopping 9.5 score (which is equivalent to getting a 27/10 on IGN). I’m pretty sure it will be the PC VR game of the year (unless What The Bat 2 comes out :P)
- ‘Half-Life 2: Episode 2’ VR Mod has been released on Steam, for the joy of the PCVR community
- Joyway has modified its previous canceled game Outlier into a new game called Dead Hook, which will be released on Quest on May 18th
- VR Skater, the skateboarding sim for PC VR headsets, is coming to PSVR 2 this summer
- Paradiddle is developing a very cool AR mode that lets you play the game while playing your real drum. This is a very good idea to make people learn to play an instrument
- Carbon Studio announces Crimen — Mercenary Tales, a game coming in May. It is an “action-packed, gory arcade slasher” set in the 17th century.
More info (Gorn)
More info (Fruit Ninja VR 2)
More info (Creed)
More info (Vertigo 2)
More info (Half Life 2 Episode 2)
More info (Dead Hook)
More info (VR Skater)
More info (Paradiddle)
More info (Crimen)
Other news
I just discovered that some senior VR devs at Meta earn 1M dollars a year. I guess in my life I made the wrong job decisions…
Praydog’s Universal Unreal VR Injector will automatically inject VR into many Unreal-based games. Here you can find an exhaustive list of them
According to a recent survey, only 4% of American teens use VR daily. We hoped for a bit more
News from partners (and friends)
Vote for Oxymore at the Webby Awards!
The VR musical experience “Oxymore” I contributed to has been nomineed for the Webby Awards. Please vote it to make it win!
Tennis League VR is coming on April 20th
If you love tennis and virtual reality, you had better know that the game Tennis League VR developed by my friends at Anothereality will release on the main Quest Store on April, 20th. A new trailer unveils its career mode and arcade mode. Wishlist it now!
Some XR fun
The dream job doesn’t exis…
Funny link
The future is bright with ChatGPT
Funny link
VRChat goes to mobile!
Funny link
The end is near
Funny link
The DecaGear is finally going to ship! Oh no, wait…
Funny link
I have a big nose. You have a big nose.
Funny link
The Oculus Quest you can afford when you are poor
Funny link
Donate for good
Like last week, also this week in this final paragraph I won’t ask you to donate for my blog, but to the poor people that are facing the consequences of the war. Please donate to the Red Cross to handle the current humanitarian situation in Ukraine. I will leave you the link to do that below.
Let me take a moment before to thank anyway all my Patreon donors for the support they give to me:
- Alex Gonzalez VR
- DeoVR
- GenVR
- Eduardo Siman
- Jonn Fredericks
- Jean-Marc Duyckaerts
- Reynaldo T Zabala
- Richard Penny
- Terry xR. Schussler
- Ilias Kapouranis
- Paolo Leoncini
- Immersive.international
- Nikk Mitchell and the great FXG team
- Jake Rubin
- Alexis Huille
- Raghu Bathina
- Chris Koomen
- Cognitive3D
- Wisear (Yacine Achiakh)
- Jennifer Granger
- Jason Moore
- Steve Biggs
- Julio Cesar Bolivar
- Jan Schroeder
- Kai Curtis
- Francesco Strada
- Sikaar Keita
- Ramin Assadollahi
- Juan Sotelo
- Andrew Sheldon
- Chris Madsen
- Horacio Torrendell
- Andrew Deutsch
- Fabien Benetou
- Tatiana Kartashova
- Marco “BeyondTheCastle” Arena
- Eloi Gerard
- Adam Boyd
- Jeremy Dalton
- Joel Ward
- Alex P
- Lynn Eades
- Donald P
- Casie Lane
- Catherine Henry
- Qcreator
- Ristband (Anne McKinnon & Roman Rappak)
- Stephen Robnett
- KaihatsuJai
- Sb
- Enrico Poli
- Vooiage Technologies
- Caroline
- Liam James O’Malley
- Hillary Charnas
- Wil Stevens
- Brian Peiris
- Francesco Salizzoni
- Dimo Pepelyashev
- Alan Smithson
- Steve R
- Brentwahn
- Pieter Siekerman
- Simplex
- Matias Nassi
And now here you are the link to donate:
Support The Red Cross in Ukraine
(Header image by Meta)
Disclaimer: this blog contains advertisement and affiliate links to sustain itself. If you click on an affiliate link, I'll be very happy because I'll earn a small commission on your purchase. You can find my boring full disclosure here.
