All the most important AR and VR news from Facebook F8

On May 1st-2nd, Facebook has had its annual F8 conference. There it has presented a lot of news regarding all Facebook companies (Messenger, Instagram, Facebook, etc…) and of course, among these pieces of news, we had some AR and VR ones. If you missed some of them, don’t worry because I’ve got you covered and in this post, I’ll summarize all the most important XR news from the F8 event.

Honestly, if you haven’t followed the F8, I have to say that you haven’t missed that much: some months ago, Facebook executives like Hugo Barra announced that they would have announced a disruptive AR/VR news at F8 and I also heard the rumor of a new AR/VR device, but in the end, we had only some predictable news, some interesting updates and some teases of the work of the R&D department.

https://twitter.com/boztank/status/966386619845455872?ref_src=twsrc%5Etfw

I came out disappointed by what happened at F8. Someone says that due to recent Facebook problems with the data privacy scandal, they preferred to delay the launch of new products to a future event (maybe OC5?), but honestly, I don’t know if this is the truth. In any case: Zuck, I’m disappointed, remember that.

Ok, and so, with much disappointment, let me tell you all the news.

Oculus Go launch

We all knew that Oculus Go would have been launched at F8 and this has been the case. What I did not expect was that the launch was going to happen immediately: I expected an announcement like “on May, 15th 2am PST we’ll open the preorder of the Go!”. Actually, Go immediately became available during the conference. And not only online on Oculus website, but also on Amazon and Best Buy supermarkets: before the conference, lots of people started posting on Reddit the pictures of the Go being already inside Best Buy shopping centers. Furthermore, all people attending F8 have received a free Go from Facebook (damn, I wasn’t there).

Best Buy is ready for the Oculus Go launch tomorrow (x-post /r/OculusGo)
byu/Heaney555 inoculus

I have preordered mine immediately: I have taken a 32GB model for 219€ (this is the price here in Italy… in the US it is $199). As soon I’ll receive it, I’ll let you know my impressions with a hands-on review. It seems that it should arrive in a week, so Oculus is becoming pretty fast in shipping devices.

I’ve purchased my Oculus Go!

After the release, we started seeing some interesting material being created around the Go. For instance a redditor showed the full unboxing of the device

Oculus Go unbloxing

while Palmer Luckey decided to steal the job to iFixit and disassembled completely the Go, showing to people how it is internally made and also suggesting that disassembling it you can substitute the battery with a more performing one (if you’re able to mount again the device…)

pic.twitter.com/R9138L0w5j

— Palmer Luckey (@PalmerLuckey) May 2, 2018

People also started making questions to John Carmack regarding the Go: Oculus instructions suggest not to use the device while charging but John clarified this is not because of risks for your safety, but just for comfort (not to have a cable attached) and for heating issues. Furthermore, while at the moment there is not the possibility to stream the Go content on an external device, Mr. god Carmack said that he’s working on a streaming feature towards the companion app.

I made a demo of that, but it will take a little while to get it integrated into the companion app.

— John Carmack (@ID_AA_Carmack) May 2, 2018

The headset comes with some plastic pieces to add prescription lenses to your Go: Oculus suggests buying VirtuClear ones for the Oculus Go (thanks Upload VR for spotting that). If you don’t want to buy prescription lenses, you must know that the Oculus Go is glass friendly and also features a Glasses Spacer to let you insert your glasses in a more comfortable way.

I’ve started reading reviews on the Oculus Go on my favorite magazines, that is Upload VR and Road To VR and I was surprised to read that both journalists expressed a quite negative opinion on the device. Especially my reviews-super-hero Benjamin Lang has destroyed completely the Go in his review, saying that it could have sense some years ago, but now it is just a step back from current technology. Furthermore, he highlighted how the app store currently lacks interesting titles and the announced “more than 1000” apps are mostly low-quality ones. David of Upload VR instead highlighted how the battery life of only 2-2.5 hours (2 hours of gaming, 2.5 of videos) is too short to do something meaningful. I have still to try the device, but this initial reviews surely don’t excite me.

Oculus Half Dome Prototype

The second most important XR news from this F8 has surely been the reveal of the Oculus Half Dome prototype: a tethered headset with varifocal display (that lets you see sharply the objects that in VR are close to your face, for instance), 140 degrees field of view and eye tracking. You can see the announcement in this video

It is very interesting to see this device and also how it works, with the display that moves inside the headset to change the focus. Everyone got super hyped, and I also really liked to see this announcement, but actually, I’ve not been impressed.

The reasons? Well, first of all, they just made a tease showing some photos and some videos: there was no device on the stage and no one has been able to try it. This means that this project is nowhere close to launch and will need at least 1-2 years to arrive at the market. This has also been highlighted on Reddit by Nate Mitchell. Furthermore, we all know that R&D departments of companies have products that are at least 2 years ahead of the market, but a research project is completely different from a product: a product has to be polished, must pass certifications, must work with currently available hardware, must have a reasonable price. The Half Dome is a prototype and not a product: to me is like they had said: “we’re working on cool stuff for the future” and nothing more.

And then, I remember well when Mary Lou Jepsen said that at Oculus R&D she worked on a functional AR/VR glass with 180° FOV, eye-tracking and foveated rendering. Read that again: those are amazing features, that at the moment we can only dream about. After having heard that, getting to know about a 140° FOV VR only device is not that impressive anymore.

The good news of the Half Dome is that it means that Oculus is still working on a PC version of the Rift for the future: it is not clear if the Half Dome will become the CV2 or the CV3, but it is sure that the Rift will live long to provide premium features to gamers and people wanting top-class VR. This is something that I predicted and it is also in line with the vision of the competitor HTC that says that standalones are products for consumers while PC VR is something for gamers and prosumers.

Other research projects

Alongside the work on the Half Dome, they also showcased very interesting works in various research areas, like:

• Markerless hand tracking: mixing computer vision and artificial intelligence, Facebook/Oculus has been able to create a markerless and gloveless hand tracking technology. The system can track the fingers even in complex contexts, like the hand interacting with an object or the two hands interacting together. The video is surely impressive and demonstrates how the acquisition of Nimble is being useful for Oculus plans: they claim to have the best hand tracking technology out there. This work makes me understand why Leap Motion has changed completely business model and is not selling the second version of its device as an accessory: soon all headsets will integrate hand tracking out of the box.
• Markerless full body tracking: Oculus is now able to track completely your body using only the RGB camera of a phone. Something that some years ago was possible only with a big Kinect device attached to a desktop PC, is now possible using just a phone. That’s impressive. Of course, the quality of the tracking is not the one of the Kinect yet, but we’re getting close. This will be very important for VR since it could allow sensorless full-body virtual reality, something that Facebook is actually working on (maybe they’ll realize my vision of Immotionar);
• Advanced avatars that resemble completely the user and with realistic facial expressions in all situations, even when the user is talking. Thanks to a camera attached to the nose gap it would be possible to track the facial expressions of the user and integrate them in VR;
• Environment reconstruction starting from multiple photos: if you have multiple photos of an environment, Facebook’s algorithms can reconstruct a rough point cloud of it. This means that with just some photos of the room of your childhood, you would be able to create the 3D model of it, enter it in VR and relieve your memories. At the moment the reconstructed point cloud of the environment seems a pointillism picture, but from it is actually possible to extrapolate a 3D model;

  

• Environment reconstruction using a phone: if you have a phone with dual cameras, you can take something like a huge panorama picture of the environment you’re in by moving your phone and in the end, you get a completely reconstructed 3D environment. This reconstruction is so well made that in a video it is hardly distinguishable from a real video: even the mirrors are correctly mapped. And all of this just by using your phone. And here, again, you could enter this environment in VR later on and re-live the sensation of being there. I found this research really jaw-dropping: spotting the difference between the real video and the reconstructed one is almost impossible.

Again, this is all research stuff, that we’ll see integrated into actual products in years. But having done this small trip inside Oculus R&D has been fantastic to dream a bit.

Social VR advancements

Facebook is the king of social networks and it can’t avoid considering the social aspect of VR. At previous F8, it announced Facebook Spaces and in this edition, it announced a lot of social VR experiences or upgrades to existing experiences:

• Facebook Spaces now lets you import the artworks you create in Quill and enjoy them with your friends. While for people like me that can’t even draw a cube this is not interesting, for artists this is awesome. Do you remember all the fantastic artistic animations created by Goro Fujita using Quill? Well, this means that you can insert them into Facebook Spaces and live them from the inside with your friends. And maybe you can also take selfies with dragons;

• Starting this month, Oculus Venues, the application that will let you assist public events (concerts, sports matches, etc…) in VR, will be available for Gear VR and Go. This means that for instance you could go to a football match in VR and you would see yourself in a stadium together with all the other thousands of people using Venues to watch the same match. All of you would watch the streaming of the match together and this would resemble having gone to a real stadium. This is a very interesting idea. Oculus has already partnered with NextVR to bring awesome content to Venues;
• Oculus Rooms is getting a big update and will let you play tabletop games, watch movies and listen to music together with your friends while you wait to start a multiplayer game together. Considering that Oculus is partnering with Hasbro to port popular games like Monopoly and Trivial Pursuit in VR inside Rooms, Rooms is really becoming a nice lobby where to hang out with your virtual friends;

  It is also nice that when in Rooms someone of your friends enters a 360 video, you can see him inside a big bubble showing the preview of the video;
• Oculus TV will feature a big screen in VR where to watch streamings from Netflix, Hulu, Facebook Video, etc… with your friends.

  

Instagram and Messenger get AR features

Do you remember Facebook Camera, the augmented reality framework of Facebook? Well, it has been ported to both Instagram and Messenger.

On Instagram, this means new 3D AR filters that you can apply on the environment or on your face to have better content to share with your friends. Some of this AR filters will be created by brands and celebrities… and if you’re thinking that this is very similar to what Snapchat is doing… well, you’re right.

On Messenger, instead, this means creating applications embedded inside Messenger that are powered by Facebook AR engine. In a great example shown by Facebook, you can see how it is possible for instance for a girl to share on Messenger the Sephora AR experience with a friend of her. She can tap on it to open it inside Messenger and have the possibility to try some Sephora make-up on the fly directly on her face using the frontal camera of the phone. She can also share some pictures of her with the Sephora lipstick on with some friends to get advice. When she’s done and has selected the makeup that fits her the most, she can directly buy the Sephora makeup, and all of this without never leaving the Messenger app. Another app lets people see cars in AR on the road in front of them so that to see a car before deciding if buying it. This opens a lot of possibilities for brands, to create AR experiences that people can share and use.

Other news

Some other interesting news:

• Oculus is updating its React VR framework and is completely re-shaping it. It has also been renamed as React 360, to underline the fact that it is a framework to create WebVR experiences mostly to show 360 photos, maybe in a tour to showcase a house or something like that;
• Facebook has partnered with RED Digital Cinema to create volumetric VR cameras. Do you remember the Surround360 camera reference designs that Facebook created to let people record 6DOF videos? Well, finally Facebook has found a partner that will create the first ones and this is RED. RED is already famous to create super-high-quality cameras for the cinema and this partnership may lead to a fantastic camera to record VR movies. Of course, it will be a very expensive professional one that will be used in enterprise environments. You can have a peek at the quality of the depth perception of a similar camera by looking the below teaser video. Videos where will be able to move in are coming!
• Oculus Gallery: Go users will found from day 1 the Oculus Gallery app that will let the user stream photos, videos and such from your phone, a local home media server, Facebook, Instagram or Dropbox directly to your Go device so that you’ll be able to see this content in VR.
• Soon it will be possible to see the 3D models shared on your Facebook feed in AR using Facebook Camera by just using a simple tap on the screen of your smartphone;
• 3D Photos: Facebook wants to help you in creating photos that seem a bit 3D

Huh, Facebook is also working on “3D Photos” for News Feed, kind of hard to tell from the keynote screen but they look like slightly more immersive photos, something between a regular pic and a 360-pic #F8 pic.twitter.com/TymRLR5SrI

— Karissa Bell (@karissabe) May 1, 2018

Full Keynotes

If you have some free time, here you are the videos of the two main keynotes

Hope this super-summary have helped you in not missing even a single AR and VR news from the F8. If this is the case, subscribe to my newsletter or share this post on your social media channels to show your appreciation for my work 🙂

(Header image by Facebook)