Ultraleap Gemini is hands tracking at its best

Today Ultraleap has officially released Gemini, the fifth-version of the hands tracking runtime. I have tried it and I’m going to tell you my impressions about it!

Ultraleap Gemini

Ultraleap is the company born from the acquisition of Leap Motion by Ultrahaptics and it has always worked towards offering optimal hands tracking solutions for the XR market. It offers both camera sensors to perform hands tracking (Leap Motion Controller and Stereo IR 170), and the engine that actually takes the data from these sensors and transforms it into actual hands data. Leap Motion has born as a solution for PC, but now Ultraleap software works both on Windows PCs and on Android VR systems, thanks to the integration inside the Qualcomm’s Snapdragon XR2 platform. Ultraleap hardware+software solution is already integrated into high-quality VR headsets like Varjo XR-3 and the upcoming Lynx R1.

Today Ultraleap has announced the official release of the Gemini runtime, the fifth generation of its hands-tracking engine that can work with the already existing Ultraleap tracking cameras. Interested developers can try it by accessing it on its official page.

According to the company, these are some of the improvements included in this runtime:

1. Improved two-handed interaction
2. Faster initialization and hand detection
3. Improved robustness to challenging environmental conditions
4. Better adaptation to hand anatomy

Ultraleap has also made significant changes to the tracking platform to be able to extend hand-tracking to different platforms and hardware.

You may recall that actually I already featured Gemini on this blog some time ago: well, that was the preview of the software, while now the official version has been released. And of course, since I believe press claims up to a point, I wanted to go hands-on with the new runtime to see if it is as cool as Ultraleap claims! (Spoiler: it is).

Ultraleap Gemini tracking

I have tried Ultraleap Gemini tracking with my Quest+Link and the Ultraleap Stereo IR 170 camera. I have used the latest camera because it maximizes the tracking FOV, so it offers a better experience for the user… but in case you want to try the runtime too, an old Leap Motion controller should do the trick as well.

I want to start my analysis of the tracking with the things that do not work on Ultraleap Gemini + Stereo IR 170 tracking:

• The detected pose of the hands and fingers is not perfectly equal to the real one, but there may be some little discrepancies (e.g. the fingers are bent with a slightly different angle)
• Pinky and ring fingers are sometimes misdetected the one for the other

• Very weird hand poses (e.g. fingers all interweaving the ones with the others) are still not detected properly
• Two hands interactions make the tracking accuracy to degrade

• If the hands are too close to real objects, the tracking quality degrades
• If you are close to the window on a sunny day, the hand irradiated by the sun has some more tracking problems, like it takes a bit more time to initialize the tracking
• If the hands move too fast while doing the punch gesture, the tracking may be lost of a while

• If you do the pinch gesture with the thumb occluding your index finger from your point of view, the gesture doesn’t always get properly detected. This can be an annoying problems when you are using hand tracking to interact with your VR experiences

If we remove all these minor issues, well, ALL THE REST WORKS. Really, it’s impressive how the system works well:

• The tracking is fast and fluid
• Single hand-tracking is super-reliable, with just some little issues with pinky and ring fingers

• Bimanual interactions are possible, and you can actually even cross completely the fingers of the two hands, or you can put one hand on top of the other, and the tracking still works!

• The hand are tracked as soon as they enter the field of view (this is possible thanks to the wide-tracking FOV of Stereo IR 170)
• The fingers tracking quality is good also in the periphery of the vision. This means that you can just use your hands naturally, without having to put them in front of your vision to see them tracked reliably
• You can move the hand very fast if you are doing the “five” gesture and the tracking still works. This opens new scenarios in using Ultraleap for fitness and rehabilitation

Finally, we have a truly usable hand tracking solution. You can use your hands naturally, without worrying too much if the hand is in the center of your vision, if one hand touches the other, if you are moving one hand too fast… in any case, the tracking keeps working. With this system, you can have proper bimanual interactions and a totally natural interface in virtual reality. If you are used to Oculus Quest hands tracking, I can tell you that this is so much better: it is superior for one single hand, and totally better with two hands, considering that on Quest, as soon as the two hands touch, the tracking stops.

Hands-on with demos

I don’t want to waste your time with videos of all the Ultraleap demos that I’ve tried since they are similar to the ones that I’ve played when I tested the preview (that article was already very detailed and there is no need for me to repeat what I have already written there). But again, I underline how my tests showed an impressive hands-tracking solution.

During my time with the Blocks demo, as long as I didn’t occlude my index finger with my thumb, I could generate easily cubes and other geometrical figures wherever I wanted in my field of view, I could use with one hand the menu that I had on the other… and I have been able to juggle a cube with my two bare hands!

I have also tried the new Ultraleap XR Launcher, an application (not available for the public, at the moment) that is like a Hub of everything about Ultraleap, containing all the demos, all the images, videos and settings in a single place controllable with hands-tracking. With this demo, Ultraleap confirmed its great expertise in designing hands-tracking UIs, and I found it incredibly natural to navigate its interface: it was for me like operating with a touchscreen in real life.

The best moment for me was when I tried the virtual hands-controlled keyboard. Thanks to the solid tracking and the visual cues on the keyboard, that showed me what key I was going to type with my finger before I touched it, I was able to type flawlessly in VR my name “Tony Skarredghost” (Actually I have written one “a” more, but that was my fault) even without haptics. Of course, being someone that can touchtype, I found it very slow to use compared to a physical keyboard… but again, I was impressed by its accuracy and its usability.

Final impressions

Ultraleap tracking is still not the perfect hands-tracking that we dream of, but it’s the best solution of this kind actually on the market. There are some glitches here and there, but all in all the system works as expected and it already allows people to use their hands in XR for natural interactions. I was sincerely impressed by its performances. It just needs a bit of more stability in some edge situations, like the hands moving fast, or the hands auto-occluding, to be able to replace controllers.

I invite you all to try Gemini at this link and judge it by yourself.

Special thanks

Apart from the technical results, I have also to say that Ultraleap is one of the best companies I have interacted with as a blogger. Today, to celebrate the launch of the new Gemini runtime, they have also sent me a swag pack with a nice T-shirt, a thermos, some stickers, and a very kind letter! They have been so cute.

Thanks, Ultraleap people, and keep up the great work!

(As a disclaimer, the swag was a free gift and they have asked nothing in change for it, not even me writing this article)

And that’s it for Ultraleap Gemini… another XR product of this crazy October… which has not ended yet! What do you think about it? If you have any questions or feedback, feel free to write it here in the comments below!