Orbbec now offers skeletal tracking: markerless full body VR is coming?

One of the reasons my VR startup failed has been because we relied on Kinect to perform full body tracking, but Kinect has start dying in 2015 and now the project is something like a zombie. We were looking for a substitute and we put our trust on a little startup called Orbbec, who started an IndieGoGo campaign some years ago to create a new Kinect-like hardware and now it’s continuing following its vision.

Orbbec is awesome in the sense that they’ve managed to create a version of what Kinect could have become if Microsoft had continued to develop it: little, smart and wireless. And powerful. It is the only Kinect alternative out there having a depth range of 8.0m (Kinect was around 9.0m): all other competitors have far smaller performances. The Persee version is wireless in the sense that it can stream its data via Bluetooth or Wi-Fi. Inside it has a little Linux operating system that takes the data from the sensors and streams it wirelessly. Super-cool! Otherwise, if you don’t mind about cables, you just take the Astra version and spare some money.

Advantages on Kinect is that it is surely smaller, more user friendly and more elegant: Kinect v2 is very big and has a mess of cables, while this is just a little box you can put everywhere. Disadvantages are price ($240 for the wireless version, $150 for the wired one.. while Kinect+adapter now costs $140) and precision of tracking (Microsoft has used a training set with bazillions of people, I don’t think they’ve had a similar training set). But Orbbec is an alive project, while Kinect is destined to die…

Orbbec has had the skeletal tracking features as “coming soon” for months. We were waiting for it to integrate it inside our ImmotionRoom full body VR system. Now that Immotionar has just shut down, obviously they released the news that they’ve implemented it.

You know my vision about full body virtual reality: it will surely be the future. We can’t continue to be in a virtual world without having our legs and our feet in VR. Not even having our elbows reconstructed with inverse kinematics. I just want to be myself in VR, with all my body, that is exactly as my real body. I want to kick stuff.

We tried offering all of this using Kinects, but each Kinect v2 is expensive, plus it requires to be connected to a PC (a PC for every kinect) and this raises the costs a lot. Each “station” of ImmotionRoom costed something like $500 (when we started, costed something like $800, due to higher prices of PCs and Kinects). To track a body at 360-degrees, you need at least 3 Kinect stations… 4 if you want a good tracking (this is from our experimental tests: a Kinect tracks correctly a body only if it is sufficiently frontal to it).

So you needed at least $1500 to try full body VR, plus your VR system (e.g. GearVR headset). With Orbbec, for the same experience, the price is half: you just need $720. Plus, being little, standalone and wireless, you don’t need all the cables and the mess required by the various Kinects stations.

$720 is a fair price for a full body VR system, considering that suits like PrioVR costs $1200-$1500. And in this case, you don’t need to wear anything! And as a super-plus, you also have multiplayer out of the box: the kinect-like system (if tracking has been made smart enough to handle occlusions) can handle various people in the tracking area, so you even enable local multiplayer. If you’re thinking that Vive trackers are cheaper… well, they are… but it is super-boring to wear trackers on your body every time you want to play at your home. Furthermore, they detect position of your body joints through IK and that’s good for playing, but for rehabilitation purposes having a precise body joints tracking is fundamental. Plus you don’t have multiplayer. (But you have out-of-the-box SteamVR compatibility, I know)

I envision the future like this: we use setups like the one of Vive or Rift, but instead of having dumb point-tracking cameras, we have 2-3 smart wireless tracking cameras we put on our walls (technically Oculus cameras are able to detect images, but they’re used only as point trackers, now). So we install little kinects instead of the LightHouse stations: these will track the bodies of all players and will let us play full body VR with our friends. Being also RGBD cameras, they could also scan our body and let us be completely ourselves in VR… something that will be enormous for social VR. Reverend Kyle pointed out that current VR avatars are all thin, all equal and that he wants to be fat in VR… well, this system of multiple cameras could scan you and let you be fat in VR, too… exactly as fat as you! With Kinect v2 this was quite impossible… with Orbbec Persee, a scenario like this can start becoming possible: they’re little and can be put easily on the walls… a bit like Oculus tracking sensors or LightHouse stations.

Of course we have been not the only ones having this idea… and that’s why VicoVR uses Orbbec depth sensors for full body VR since months (technically they have made a custom sensor based on Orbbec depth sensor). If you’re wondering how VicoVR can use Orbbec skeletal tracking before it has been released… well, it is because 3DiVi, the company behind VicoVR, is the one that has implemented the skeletal tracking for both systems (I guess that one day these companies could become a single one… they’re too close each other…).

What still lacks from my vision of the future is:

• Price: it is still too high. Not many people would spend much money for a full body VR system. I mean, when pitching our ImmotionRoom system, lots of people said us “Why do I need full body VR? I’ve hands controllers… I don’t care about having my feet in VR”. And asking them to spend $1500 (+ a lot of configuration hassle) to have it was too much. I think that a full body VR system to become popular has to be under the $300 price tag now… and integrated into the headset hardware in the future. 3 Orbbec costs too much now. VicoVR costs $399 and uses only 1 sensor, so I guess it can offer only pseudo-frontal tracking and looking their demo videos confirms it. But true full body VR is 360 degrees, so you need more than one;
• Performances: I’ve not tried VicoVR/Orbbec/3DiVi tracking (if Orbbec or VicoVR want to send me some kit to review… I would be very happy!), but I can guess that due to training data amount, Kinect v2 is still superior (and the body joints count confirms it… 19 body joints for 3DiVi, 25 for Kinect). Framerate of data is 30FPS, as Kinect… and wireless surely adds some latency. I can assure you that Kinect tracking was not suitable for VR: there was lag of data, hands and feet were very unstable, joint position was not precise. We had problems in making people interact with our system, due to all these problems: for example, we tracked the head position to offer room scale, but the tracking was imprecise and run at 30FPS and this lead people to motion sickness. As you can see from the below video, hands tracking with Kinects vs Oculus Touch shows that hands interactions with Kinects is not that good. We need something with millimeter accuracy and at least 60FPS. Surely VicoVR guys have made optimizations, but I don’t think they can compete with Oculus Touch, too;

• Cross-platformness: we made ImmotionRoom to work with all available headsets, both wired and wireless and with all available sensors (Kinect v1 and v2). Some framework like ImmotionRoom, but with Orbbec and VicoVR sensors, is needed. Something that supports multiple sensors and multiple headsets. VicoVR only works with GearVR and Cardboard/Daydream, for instance. We don’t need full body VR fragmentation, we need a common framework;
• Content: content is the biggest issues of VR nowadays… and these innovative solutions are even in a worse situation. Some months ago I was reading a VicoVR review on the web and the article was something like “Yeah, super cool… but why developers should develop for such a little niche?”. We had the same problems… no one was interested in making a game for a userbase of 100 people. Only interested customers were business ones, to make installations somewhere, maybe for marketing purposes (full body VR has always a great success in exhibitions, I can assure you!). VicoVR has made some little fun games, but people are even complaining about the number of Oculus compatible games (and we have masterpieces like Robo Recall)… so surely some games aren’t enough.

Anyway, I’m sure that the full body VR future, one day, will happen. But there is still a lot work to do. For now I just appreciate the news… a new body tracking sensor that is alive, kicking and updating is something that will enable lots of new experimentations, both in VR and outside of it. I make my compliments to Orbbec, 3DiVi and VicoVR for their hard work… and because they still believe in body tracking as fundamental in lots of applications, especially in virtual reality! Full body VR rocks!

I’m waiting to see your opinions about it!

(Header image by Orbbec)