Hands-on: Immersix may revolutionize eye tracking with its accuracy

I realized I never told you about a very interesting startup I met in May at the VR/AR Expo China in Shanghai: it is called Immersix and is bringing a slew of innovations to the eye-tracking field.

Immersix and its retina tracking

Immersix is a Western startup focused on building eye-tracking solutions for XR. Its big innovation is that instead of trying to track the pupil like almost everyone else is doing, it tries to track the retina of the user.

According to the company, tracking the retina has huge advantages over the other solutions:

• The retina is unique for every user, so it is possible to use it for authentication and is very difficult to spoof
• The retina almost doesn’t change over the years, so you can perform the eye calibration only once and then use it forever
• The retina is the ground truth of the eye, so it doesn’t matter if the glasses move on your head while you use them. The once-in-a-lifetime calibration is still valid even if you remove and put on the glasses again
• The tracking can be very precise
• The module offered by the company can be very small (just two tiny modules close to the nose), consume very little power (less than 10mW), and be very fast (up to 120Hz).

It seems that Immersix can solve almost all the problems of the current implementations of eye tracking in MR headsets. For instance, I find it pretty annoying that after a few times I put a headset on and off, the eye tracking performance degrades and I have to perform the boring calibration again. With this solution, it shouldn’t be necessary anymore.

How does it work?

Well, the idea is very interesting. Basically, the module for each eye is only comprised of a camera and an IR LED to illuminate the eye. Notice that one LED is sufficient, so the module can be smaller than many eye tracking solutions that need a full circle of LEDs.

During the calibration stage, the user is requested to perform some eye movements. Every camera module looks at the eye, but instead of focusing on the exterior part of it, it tries to look INTO it.

The camera so gets the data of the retina over time. The calibration is studied so that the various movements of the eye let the camera see step-by-step the whole retina. The software can then reconstruct the full retina image and extract a full fingerprint of it. The software calibration basically reconstructs a complete feature map of the retina of the user. And since the retina doesn’t change over time (until you become very old, at least), this map can persist for many, many years.

After the calibration, the system can actively track the eyes. When this happens, every camera still looks at its eye, and in particular at the region of the retina that is possible to see with the current eye orientation. Using a special algorithm ideated by Immersix, it is possible to find the exact eye orientation by matching this visible small retina region to the whole map of the retina reconstructed during the calibration stage.

According to Immersix, this tracking is very precise (they claim sub-degree accuracy). And since the camera doesn’t track the pupil position related to the glasses, but tries to track the absolute eye rotation with regard to the retina, the retina acts as a ground truth; the tracking is so insensitive to movements of the cameras. You can so put the glasses on and off, and the eye tracking keeps working without any recalibration, even if they are in a slightly different position and orientation every time.

Hands-on Immersix eye-tracking

Ok, now you are probably wondering if this works as claimed. Well, I’m writing an article about it… so you can already guess the answer (It would be cruel to write an article saying “hey, this startup has this fantastic idea… but you know what? I tried it and it hugely sucks”). Let me tell you how my hands-on experience went to answer your question.

A representative of the company let me go backstage to try his solution. He didn’t let me record anything, so I will use some videos and images provided by the company to help me convey the experience I had.

I was handed some frames of glasses with the Immersix technology installed on them. They were not smartglasses or AR glasses, just frames to test the tracking technology. I could clearly see the circuitry, so I was not testing a product, but a prototype. This was clearly a tech demo with a devkit.

After I had worn these special glasses, they were connected to a laptop. At this point, the eye-tracking calibration application started. I could see on the laptop screen some big markers (probably ARuco Markers) and in the middle a small red rectangle. The markers were clearly there to let the prototypical glasses track the screen position: since this tech demo couldn’t do the calibration on the device (as I’ve said, there were no AR or VR, just the frames), it had to do the calibration on the laptop, so the glasses had to know where the laptop screen was. I guess that if this software were installed directly on the glasses, these markers wouldn’t have been necessary.

Anyway, there was a tiny pointer associated with my eyes, and by moving my eyes, I could move this tiny pointer. My task was sweeping this pointer on the screen to fully erase the small rectangle. It was like trying to remove a rectangle in MS Paint by using the eraser tool with a small radius. I had to repeat this procedure of moving my eyes back and forth to erase the rectangle a few times. I had the impression that the more I went on, the more the eye pointer was “lagging” behind, so I had to move the eyes more left and right to erase the small rectangle. Probably this was made on purpose to make me move my eyes in a more lateral position, so that the cameras could record all the regions of my retina.

The calibration lasted a few minutes. It was easy to understand, but it was a bit longer than I hoped for, and also left me a bit of eye strain from the last moments where the pointer was harder to move. Anyway, nothing dramatic; I went through it pretty easily.

After that, my eye tracking could start. To show me that my eye tracking was working, the person from Immersix launched another app, and I could see another set of ARuco markers, plus a grid of red circles on the laptop screen. I could look at one of those circles to make it blue. This was proof that the eye tracking was working. And I have to say it worked pretty well. The accuracy was definitely high, and it was always changing the color of the element I was looking at. Pretty cool.

Of course, I could not verify the claims of the sub-degree accuracy or the 120Hz framerates. But I can easily testify that the system works as advertised, at least for a menu similar to the one of the Apple Vision Pro. And that’s already a great result, because some of the current eye-tracking solutions even fail with this type of menu.

I also could move the glasses a bit and check that the system was able to work without any recalibration. Also did worked pretty well in my quick tests.

Final impressions

I was really surprised by the solution offered by Immersix. It’s a new way of doing eye tracking that could remove the problem of calibration and would allow the creation of eye tracking modules that are small, consume little power, and are accurate.

My hands-on experience proved it is effective. I had just a few minutes to try it, so I could only run a tech demo and not run extensive tests on an actual AR device. So my judgement should be taken with a grain of salt. But my test with it was definitely positive, and everything worked exactly as advertised.

Probably my only concern is about privacy. This solution has to fully scan my retina, and this means that someone should store the full data of my retina. But this problem could be solved by saving the data locally, so it never leaves the user’s device.

I think it is a very promising product, and if I were a company making headsets and glasses (e.g., Meta, Pico, etc.), I would definitely have a call with these guys and evaluate their solution. Evaluation kits are available, and you can reach out to the company through its website to ask for them.

(Header image by Immersix)