How to use Kinect with HTC Vive

When I talk about the full body virtual reality system ImmotionRoom that we’re developing at Immotionar, the first question that people asks me is always “Why Kinect? Kinect is dead”. I’ve already answered to this question, explaining that yes, it is slowly dying, but Kinect is still the best off-the-shelf infrared body-tracking sensor that we can find on the market (due to the mega-training samples fed to its algorithm). So, while we hope that other competitors like Orbbec and VicoVR will reach its quality level, we have still to use it… and this is the same reason why lots of research projects still employ the kinect.

But then there is another important question people make me: “I’m a Vive user… and Vive and Kinect interfere a lot. I like your videos about full body virtual reality, but I will surely not be able to use it…”

Well, let’s be honest: this is true. Strangely, infrared-based tracking of Oculus Rift seems not disturbed by Kinects, while laser-based Vive Lighthouse is really disturbed a lot. Someone says that radiations with wavelengths around 900nm are the issue. This is a real problem if someone wants to use our system or is doing some research projects using virtual reality and Kinect (I know a guy working on this for rehabilitation purposes), so I want to give you some advices on how to handle this problem:

1. First of all, we noticed that things are now slightly better. Don’t know if it happened by chance, but it seems that compared to 6 months ago (when we shot the above video), Vive tracking is disturbed less by the Kinects. Maybe they’ve made the algo more stable and able to recover better from various problems. Six months ago, Vive was quite unusable with Kinects, while now I find it only annoying: sometimes the HMD screen gets grayed out and you can’t see anything, but I managed to use our system for some minutes with only few graying;
2. The real problem happens when the HMD (or the controller) directly faces the Kinect, so it is completely hit by lots of full-power infrared-rays. So to solve the issue, it’s better to put the Kinect in a position so that it never faces directly your headset. In our user manual, our advice is to put it in a lower position, like on a table, so that if you don’t duck, you have no problems, since your face is really above the sensors. Look the images below to have an idea of what I mean with “low position”. We’ve seen that also putting the Kinect high in height can help, because rays arrive more skewed to the headset;

   

3. Another thing that works is putting Kinect farther: the more your headset is close to the sensor, the higher are the chances of getting interferences. If you use your Vive with a Kinect that is far away, you’ll experience less problems. Remember that Kinect v2 can see  your skeleton up to a distance of 4.5m; and that can detect raw depth up to 9m;
4. Connecting Lighthouse towers in high position and with the sync cable helps a bit;
5. Having a Kinect with the same orientation of a Lighthouse (e.g putting a Kinect and a Lighthouse station close to each other, facing the same direction) makes the problem even worse;
6. In case of interference, putting the object in direct view with Lighthouse stations, re-set the object in sync: when a controller is getting lost, putting it in air, re-enables it;
7. Solve the problem using filters on Kinect to filter-away the wavelength causing the interference: I’ve never tried this solution by myself, but someone told me that it works, since you remove the interfering rays. You put this filters directly on your Kinect, so to block all rays with the wavelengths causing the problem;
8. Upgrade to a Vive Pro or another Steam Tracking v2 headset: all interferences problems are solved this way. (Thanks Yuanjie Wu for the tip)

These hints will not solve your issues completely, but will let you use Vive with Kinect in a satisfying way.

We’ve recently discovered that these issues hold only for Kinect v2. Kinect v1, the little brother of Kinect v2, which has worse performance, shows no problems in working with Vive. We just tested it: I tried to put my Vive headset in front of the sensor and cause an interference in any way, but it never happened. I can imagine that sometimes it may even happen, but it’s a very rare case. This made us happy, since this means that our recent Kinect v1 integration will let even Vive people to try our system happily! So, if you’re in doubt and you don’t need great accuracy, use a Kinect v1 with Vive for your research projects.

About the other sensors (Orbbec, VicoVR) we still don’t know about interferences but we’ll investigate in the future.

As always, if you have further advices, opinions, etc… I’d be glad to hear from you in the comments section! And of course I’d be also glad if you like and share this article!