Virtuleap wants to read (and train) your mind while you play in VR

Virtuleap is for sure one of the most promising startups I have talked with in this 2019. I had the pleasure of talking with the CEO Amir Bozorgzadeh some weeks ago (sorry for the delay in publishing the article, Amir!) and I was astonished by his ambitious project, that is understanding the emotions of the user while he/she is just using a commercial headset. This means that you may just wear your plain Oculus Go, without using any additional sensor, and the Virtuleap software may be able to “read your mind”.

I can read in your mind that you are intrigued, so keep reading…

How Virtuleap has born

Amir Bozorgzadeh told me that he comes from a background in market research and mobile gaming. Thanks to an accelerator program he attended in 2018, he got in contact with virtual reality and remained fascinated by it. At the same time, he started writing technical articles for big magazines like VentureBeat and TechCrunch as a freelancer (this may be the reason why you have surely already read his name).

Due to his background, he started thinking about performing analytics and market analysis in virtual reality. He got quite bored by the related solutions of the moment, since as he states, companies were just “copying-and-pasting Google Analytics into virtual reality”, while he thought that virtual reality had much more potential for the analytics sector. He thought that copying traditional tools to VR was wrong, and so he asked himself “What is the new type of data that can make analytics and consumer insights interesting in AR and VR?” and founded Virtuleap to answer this question.

Analysis of emotions in VR

AR and VR are actually great technologies to study the behavior of the users. Virtual Reality is able to offer the emulation of reality and this is great to study the behavior of the users in real situations. If you apply devices like eye tracking, skin conductance sensors, heart rate sensors or even EEG/EMG to the VR experience, it is possible to detect important information about the psychological state of the user that is living in VR.

This is of fundamental importance for psychologists, for instance, to be able to detect and cure anxious states in patients. Measuring the stress levels during the VR experiences may be useful for a psychologist to detect if the arachnophobia in a patient is still present by showing him a virtual spider, for instance. Or can be used for training to see if the trainee can adapt well to stressful situations. Or in marketing to see if the user is interested in the product he/she’s being showcased.

As you can see, the applications are really many, but there is a problem: to have reliable results, the user must have complicated setups featuring all or some of the above cited devices. And they are expensive and cumbersome… no consumers want actually to wear all of them.

Virtuleap and the analysis of psychosomatic insights

Amir noticed that all the consumer VR headset have almost no one of the above sensor embedded inside: this means that creators have no access to biometric data. So he asked himself “How can I offer biometric analytic data on the bottom layer of mainstream AR/VR devices? How can I offer biometric data on a cheap device like Oculus Go?”.

Thanks to his passion for psychology, he started so studying what he calls “psychosomatic insights”, that is what can be inferred of the status of the user from body language alone. We have years of scientifical studies on animals and human beings that relate bio-language to emotional insights and so he and the other Virtuleap founders started working on porting all this knowledge from the research environment to a viable business product.

Oculus Go is really a simple 3 DOF VR device, but actually, it can give us 16 channels of body capture. To be exact, this is the data it can provide:

• Headset Data
  • Angular Acceleration
  • Accelerometry
  • Angular Velocity
  • Rotation
  • Camera Rotation
  • Node Pose Position
  • Node Pose Velocity
  • Node Pose Orientation
  • Line of Sight
• Hand Controller Data
  • Rotation
  • Local Rotation
  • Local Position
  • Local Velocity
  • Local Acceleration
  • Local Angular Acceleration
  • Local Angular Velocity
  • Camera Controller Rotation

All these data, got from the sensors installed on the Go and its controller, may be used to track the nervous system of the user. By tracking all the movements of the users, including all his/her subliminal movements, like micro-gestures and micro-motions that he/she is not even aware about, it could be possible to infer something about his/her emotional state.

Using neuroscience research, it may be so possible to just let a user play with a plain Oculus Go and infer if he’s angry, bored, stressed or interested during all the stages of the game. This is massive. Collecting all this data over time while you perform some specific VR tasks, it is also possible to profile your brain, and so understand how you handle stress, how you are attentive, how you are good in problem–solving and so on.

The idea of Virtuleap is to offer a whole framework for emotional detection and analysis for AR and VR. In the long run, the framework will adapt on the sensors with which the XR system will be equipped: so if the user will use a plain Oculus Go, the system will just analyze the micro-gestures defined above; if there will be eye tracking on the VR headset, then the system will use the micro-gestures + eye data; if there will be a brainwave reading device, it will be used as well, and so on. The more sensors will be used, the more the detection will be reliable. But the system should work with all the most common commercial XR headsets, so that the analysis can run with consumers and not only in selected enterprise environments.

The company also plans adapting the algorithms to the kind of application that the user is running: detecting stress levels in a horror game is different than detecting it in a creative application, for instance, because different brain areas are involved or the same brain areas are involved, but with different activity levels. So, the company will tailor the detection for various different scenarios, so that to offer better emotion analysis in the experiences.

Current status of the project

The plan of Virtuleap is incredibly ambitious and I think that if Amir will actually be able to realize it, his startup has the potential of becoming incredibly successful and profitable.

But the road in front of it is still long and complicated. Currently, the theory that is possible to go from micro-movements data to a reliable detection of all the emotion of the user has still to be proven. Also because current commercial devices like Oculus Go supply noisy data for the analysis. The company has understood how to clean the data it has to analyze and what mathematical models are most promising, but it has actually just started going out from research mode to enter the production mode. So, it is just at the beginning and nothing has been properly validated yet.

What it can do now is detecting “arousal” states in the users. And no, we are not talking about THAT KIND of “arousal” (you pervert!), but of a triggered state of the brain, a spike in emotional activity of any kind. So, Virtuleap is currently able to detect when you’re getting emotional for some reasons while you’re using VR. But it is currently not able to detect why you’re getting this spike: maybe you’re scared, you’re overly happy, you’re stressed, or you’re aroused in-that-sense. I have to say that in my opinion is already something exciting (my arousal level has risen when I heard this!), but of course, it is not enough.

That’s why the company is currently looking for partners in different sectors so that to gather more data that will make the machine learning system more reliable and able to distinguish the different emotions. These partners are studied so that the “arousal detection” can be already useful for them because the experience has already a defined context. For instance, if the user is playing a horror game and there is a detection of a spike in arousal, it is almost sure that it has happened because the user is overly scared… it is not necessary to understand the emotion, because it is obvious. Currently, Virtuleap is partnering with companies in sports streaming, e-commerce, security training, and many others.

Neuroscientists know how to correlate behaviors to parts of the brain that get activated. So mixing the data of what the user is supposed to be doing with the one detected from the headset, it is easier to understand what is happening in the mind of the user.

The plan is gathering data, train the machine learning system and then validate the results using the ones from other devices like eye tracking and EEG.

The startup has an internal test app called The Attention Lab. You can enter this virtual lab environment and play little games inside it. One game that there is now, for example, shows you a pirate ship that attacks you and you have to fight it back by launching bombs at it with precise timing. This game wants to analyze how you do cope with stress: how easy you do get stressed, how do you behave in stressful situations, etc…

After that you have played the games, you can return to the virtual laboratory and see the data about your brain that have been collected. You can look at the profile of your brain, and understand better how is your personality. You can also share it with other people and challenge your friends on who is better at stress management, for instance.

At the moment, the app is for internal use, but the company is planning to make it public in the future, also to be able to gather data from more users and train its ML models better. I hope I will be able to try it soon, because I’m curious to experiment with this black magic of reading my emotions without using sensors!

During fall 2019, Virtuleap plans to already make available in beta the API, reporting and dashboard systems. So, interested companies will already be able to experiment with emotional analysis in VR. The prices won’t be high, says Amir, and mostly are needed to cover the costs of the cloud.

But when I asked Amir how much time is needed for the framework to become fully complete, including all the possible sensors and giving fully reliable data, he told me that probably 3 years is a probable timeframe. So, the road in front of this startup is still very long.

Recent pivot to training

Before publishing this article, I have got in touch with Amir again and he gave me a little scoop exclusive to this website (yay!). He told me that actually now Virtuleap is experimenting a pivot to brain training technology. That is, using all the expertise that the team has gathered in neuroscience not only analyze the brain, but also to create VR experiences that can train the brain in developing new characteristics. For instance, you can train yourself to cope better with stress, to orient yourself in new environments and so on. This is very intriguing… and VR has already proved to be very effective to train and educate the users. It has a strong influence on the brain.

The Attention Lab should serve this purpose as well, and it will be released in beta in September of this year.

Emotional analysis and privacy

When talking about the collection of emotional data, there is the obvious concern of privacy. Amir told me that his purpose is helping people that have psychological or neurological problems, but he agrees that the technology may be misused by marketers and Zuckerbergs of all kinds.

“With BCI, privacy doesn’t exist. The computer is you”. I think that this quote of his summarizes it all how BCIs have great potential but also create great risks for our lives.

In his opinion, to mitigate this issue, first of all, all gathered data should be anonymized by default. Then, every user should be informed with a clear prompt (not long documents in legalese) on what data will be gathered and from which sensors, letting him/her choose what could be harvested from his/her brain. This should help, even if the industry is going forward too fast and privacy protection is a bit lagging behind.

A final word for VR entrepreneurs

I could really feel the passion of Amir for AR and VR. I asked him a piece of advice for all people wanting to enter the industry. He told me that he doesn’t like cowboys, people that entered the field just to make money when there was the hype.

He also doesn’t like who is in the tech only in part. “You can’t stay with a hand attached to a safe branch and the other hand to AR/VR”. He advises people that if they want to enter this field, they have to burn the bridges behind and fully commit to immersive realities. This is what the industry needs and this is the only way to gather his respect.

He also added that in his opinion all this focus on VR and AR for gaming is wrong: companies should invest more money in enterprise products, in useful solutions, in standards, and in WebVR technology… that is, in making the technology go forward. Gaming is just a little part of VR, he added.

---

I really want to thank Amir for the time he spent with me (he’s really a great guy) and I wish him good luck for Virtuleap. I think that the idea is very ambitious and overly interesting, both for analysis and for training… but its success will depend on how much it will be validated by scientific verification in these months.

If you want to collaborate with Virtuleap, get in touch with Amir or feel free to write me and ask for an introduction. And if your arousal level has gone above the required threshold, don’t forget to share this post with your peers, to make my arousal level go up as well 😉

(Header image by Virtuleap)