AWE: Ultraleap’s event cameras may be the future of hand tracking

One of my highlights of AWE has been for sure Ultraleap’s booth. Not only because they are very nice people it’s always great to hang out with, but because their use of “event cameras” for tracking hand movements completely surprised me.

Ultraleap with tracking cameras

The first demo I tried when I entered Ultraleap’s booth was its flagship hand tracking with a VR headset. Ultraleap has recently released an update to its runtime and the new version, Hyperion, is an evolution of the already amazing Gemini. The demo showcased was an introductory experience to hand tracking with a cute robot, which let me wave my hands, grab objects, create objects, draw in the air, etc…

I didn’t have the time to carefully evaluate all the tracking capabilities of Hyperion, but from my first tests, the feedback is amazing. Ultraleap is in my opinion the best hand tracking on the market today and it’s always surprising how it is able to keep tracking the hands even when you start crossing all the fingers of the two hands together. It’s incredibly stable and accurate, not only with the single hand but also for complex bimanual interactions.

What surprised me was not the hand tracking quality, but the device on which this hand tracking was happening. It was a reference design by Goertek which was mounting no Leap Motion controller. All the fantastic tracking I was seeing required no Ultraleap hardware.

Until now, all the headsets on the market featuring Ultraleap hand tracking (e.g. XTAL) had to incorporate the hardware of the Stereo Controller, but this device I was trying had none. All the tracking was happening via the cameras of the headset, the same ones used for doing the positional tracking. This was amazing: I could see the same flagship quality of Ultraleap but without any need for dedicated hardware. This is a great step ahead for headset manufacturers, that can now integrate such amazing hand tracking without the need for additional cameras (and so added costs and weight).

Doing microgestures

There was a specific demo of Ultraleap that let me try the detection of micro gestures in a VR environment. Micro gestures have been studied for a few years now, but they have become pretty popular in our space lately because of the use of hand tracking that Apple is doing on the Vision Pro. It makes sense that Ultraleap shows that is able to replicate the same functionality with its software.

There were two micro gestures detected in particular: one was the “swipe” of the thumb over the index, that is sliding the thumb fingertip left or right along the index finger; the other one was the “click” that happens when you make the index and thumb fingertip to touch.

There was a demo that let me slide and click elements on various UIs using only micro gestures. I have to honestly say that the system worked, but still required refinement. The gestures were not always detected correctly: most of the time yes, but not always. And as we all love to say: no one would ever use a mouse that works 90% of the time, you need an accuracy that gets as close to 100% as possible. So this interface was not ready for prime time yet.

The “click” microgesture had good detection rates. The “swipe” was more problematic. The swipe had also an additional problem; after you swipe the thumb in a direction, e.g. to the right, your thumb has to return to its rest position, so it has to immediately swipe back to the left. If the software has not been developed to take care of this, you get a swipe left for every swipe right, which has literally no sense. Sometimes I got frustrated by trying to operate the UI of the demo with this interface and had the menus move in a direction and then immediately returning back to the previous one.

Another little issue was that I had to make my hand visible to the system, so it couldn’t stay at full rest along my body, but I had to put it a bit in front of me, so the cameras of the headset could see it. This is the same problem I had with the Vision Pro: it’s good if you are seated because you can interact with the arm at rest on the armchair, but it is a bit uncomfortable if you have to keep the hand in that position for a long time while you are standing.

The tracking of micro gestures with the VR headset was good but could have been better. And then, when I thought my tour was over, they showed me their best demo.

Hand tracking with event cameras

The Ultraleap team brought me to another part of the booth, where there was another microgesture demo, but this time happening on a sort of Frankenstein AR headset. The base device was a set of Ray-Ban Meta glasses where on the right frame was installed a device that made the frame thicker. This device featured a camera and was connected to an external small box. Ray-Ban Meta was playing some music from the phone, and using micro gestures I could operate on my playlist: with the “swipe,” I could change the song back and forth, while with the “click” I could play/pause the music.

The microgesture detection seemed more accurate in this demo. Not perfect, but much better. Also, they had introduced a “dead time” after the swipe gesture, so after I did a swipe right, the system for some time did not detect a swipe left, to give my thumb the time to return back to its initial position. My user experience was quite good and I found the demo interesting.

There was a guy working on AR R&D at Ultraleap next to me and he started explaining to me how the system worked: he said that the tracking was using event cameras and then blablabla. My brain started melting at “event camera”. What the hell even is an event camera?

What is an event camera?

I needed to show all my ignorance to Ultraleap about my not understanding at all what they were talking about, but luckily for you, you don’t have to do the same, because I have copy+paste superpowers and I’ll use them with Wikipedia:

An event camera, also known as a neuromorphic camera, silicon retina or dynamic vision sensor, is an imaging sensor that responds to local changes in brightness. Event cameras do not capture images using a shutter as conventional (frame) cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise.

Event camera pixels independently respond to changes in brightness as they occur. Each pixel stores a reference brightness level, and continuously compares it to the current brightness level. If the difference in brightness exceeds a threshold, that pixel resets its reference level and generates an event: a discrete packet that contains the pixel address and timestamp. Events may also contain the polarity (increase or decrease) of a brightness change, or an instantaneous measurement of the illumination level, depending on the specific sensor model. Thus, event cameras output an asynchronous stream of events triggered by changes in scene illumination.

From the Wikipedia page about event cameras

Long story short, an event camera is a camera that does not grab traditional frames but just detects which parts of the image have changed between the current frame and the previous one. One of the leading companies in the sector is PhopheSee and you can watch a very good explanation of how these cameras work and how they can be used in the following video:

Ultraleap is experimenting with how to use these cameras for micro gesture tracking and is investigating if it is possible to use them also for full hand tracking.

Why event cameras matter for hand tracking?

After they explained to me what an event camera was, I was genuinely curious about why taking the effort of working with this kind of device, with which you lose a lot of information (since they only work with the difference between frames, you lose all the information about texture and shapes you have in a traditional camera frame). The Ultraleap engineer explained to me that event cameras are very lightweight on resources: they can consume as little as 23mW when in use and 5mW when on standby. And since they only transmit the pixel differences between frames, which are usually small compared to full frames, they have very low bandwidth and so don’t need a powerful processor: the small box powering the whole gadget was a cheap Raspberry Pi 0 (basically a potato with a chip). And they can do all of this while going at a tremendously high framerate: as you may have heard in the video above, event cameras usually go at least at 10,000 frames per second.

What Ultraleap was trying to prove was that this may be the right technology to do hand tracking on AR glasses: it has very low power consumption, it has very low computational consumption, and it is highly reactive. It is the ideal tech to be installed on lightweight AR glasses whose battery lasts all day. I was very surprised by this approach: five minutes before I didn’t even know these gadgets existed and five minutes later I understood they may be a very valid approach for tracking in immersive realities.

In the prototype I have been shown, there was only one event camera on my right frame, pointing downwards. I guess that in a production environment, we need two, one for the left hand and one for the right one, but I have not asked this question.

After I had been revealed all the specifications of the camera, I understood why the tracking was working better in this demo: being the event camera so reactive, it was able to better detect the small movements of my thumb over my finger. Well played, Ultraleap, well played.

This use of event cameras was one of the most interesting things I saw at AWE. Anyway, just to be clear, it’s still a prototype: Ultraleap is not announcing any products using event cameras yet. But I think it will be very interesting to follow its developments and especially to see if they will be able to do full hand tracking and reach the same quality they have on their Leap Motion Controller 2 with this type of cameras.

And that’s it. I hope you are enjoying my reportage from AWE. If this is the case, do not forget to share my posts on your social media channels to help me spread the word about these very interesting projects!