Laval Virtual: Hands-On with CleverPoint and PatchXR

Two weeks ago I have been to Laval Virtual, one of the most important European exhibitions about immersive realities. I have already described to you two good experiences I had with XR Hardware (Lenovo AR glasses, and Actronika haptic suit), and today I want to close the topic Laval by telling you about two great software experiences I had there: CleverPoint and Patch XR.

CleverPoint

CleverPoint is a Polish startup building a full hardware+software solution aimed at analyzing the reaction of the user to the VR experience he is living. The data it collects may be used in many fields, and at Laval, they were showcasing their solution to analyze the cognitive skills of the user, and how he/she can properly work in a team.

I was welcome by Kirill Krasnogir, the co-founder and CEO of the company, who at first asked me some data about myself (age, weight, etc..) and then made me wear a Quest 2 headset, who had installed on the facemask some sensors that could extract EMG data about how the muscles on my face were contracting. There was also a little box on the back of the device, and I guess it was some sort of battery for this additional hardware. Kirill attached two adhesive sensors on my left hand, and they looked like the ones used for heart rate detection. I was given just one controller to use with my right hand.

At that point, the experience started, and it looked like a mix of a brain training and a relaxation app. In the beginning, I was asked to relax, then, after a short tutorial, the interesting part started: it was composed by a few short sessions (I don’t remember the exact number, but was like 7-10), each one asking to perform a certain task. These tasks seemed drawn from a brain training app. Some examples are:

• Press the trigger of your controller only if you see the name of the color written with the right color (e.g. RED written in red color was ok, RED written in green was not)
• Press the trigger of your controller only if you see the name of the color written with the right color while following with your controller pointer a dot moving on the screen
• Click on the 25 numbers in a 5×5 grid in the right order
• … and so on

As you can see, simple tasks, but which require your brain to think and coordinate with your hands. Between one exercise and the other, there was a short relaxation session, requiring you to breathe in the proper way.

The experience was quite interesting: not fun, but interesting. I like brain training games, so I’ve found it fun to play them. I could perceive my brain working in trying to find a pattern to solve them faster and better as soon as a new session with a new game was starting… I had to react pretty fast for every new game. Some games lasted a bit longer than I would have liked to: ordering 25 numbers is ok the first 2 times, but at the 5th time it becomes really annoying. But I guess that was the purpose of the experience: stressing my brain in every possible way, and seeing how it reacted. The relaxation parts served to make the brain have a break, but they worked weirdly for me: since I’m constantly working and I don’t sleep much, very soon I started falling asleep during them, waking up just when the next exercise started. Kirill told me he was a bit “worried” to see my head completely falling down in these moments, but actually, it is a completely normal phenomenon, I’m able to sleep everywhere (including the boring XR conferences about “the metaverse”).

At the end of the whole experience, who lasted around 20-25 miutes, I felt quite relaxed (I mean, it was a good experience to sleep into), and Kirill invited me to his laptop to show the report of my performances in the application. It was super-interesting that there was not just a score, but a full 23-pages-report about what are my cognitive skills and how I can perform in my daily work life. The application analyzed not only how well I performed at doing the tasks, but also what were my stress conditions while I was using the application. And it did that using the heart-rate plus muscle-contracting sensors and come out with a report of my current work performances. So they evaluated how I’m good at performing cognitive tasks, if I’m currently under burnout, if I’m able to switch fast between one task and another, how much a new problem stresses me, how good I am at solving different problems, and so on. They showed me how my brain works while do my job taking those little brain-training tasks as a sample. He showed me the report and explained every graph to me (I’ll link to you some images here below), and I could see that some things that he was saying were coherent with what actually happens when I work every day, so it seemed to me that the analysis was valid. Since it was a quick test, I can’t assure you I wasn’t a victim of the Barnum effect, but at least I can confirm to you that the numbers of the report at least had sense for what I know about myself.

Kirill explained to me that these reports are not fixed in stone for every individual, but if you repeat the tests every month, you get different results depending on how your mind evolves and your stress levels at the moment. The company offers courses to improve in the areas the report underlines you have to improve in: the solution they offer is complete, because it offers both the hardware and the software for the assessment of your cognitive skills, the report, and the courses to improve yourself with. He said that they periodically do these tests in their own company, this way the employees can improve their working life. Besides that, thanks to the reports, it is possible for them to understand what is the current status of each employee, so for instance if in this period someone is too stressed, they know he has to work less; or if he’s instead in a moment where his mind is in hunting mode in solving problems, he can be given whatever task he can solve and be very productive for the company; or maybe he hasn’t to be interrupted much, so people around him should be careful; and so on. I think this is great to improve the productivity and the wellness of every working team. It’s also interesting that Kirill said that there’s no “right or wrong result” in the report: every mind is different, and every mind should be “exploited” for its best characteristics. This is not an IQ test.

Cleverpoint is working on this solution since many years, and it is possible to see that from the great work they are doing. I also asked about the sensors they are using, and they told me that they found the use of heart-rate sensors on the hand plus the EMG on the face muscles to be the most reliable solution to obtain the results they wanted. EEG or other solutions were too noisy for the current status of the technology. They started before VR, but virtual reality has been a perfect technology to add to their solution, since with its presence it lets the user immerse totally in a simulated reality and so his reactions to the stimuli are more natural.

I came out with a great impression of this startup: what they do is pretty interesting, because it doesn’t let only people improve their cognitive skills and the ability to work in a team, but can also prevent people from burning out (thanks to early detection of the symptoms), with a great improvement to the quality of life of the employees and also a lot of saved money for the employers. And as I ve said, from what I was able to see in my quick test, the results of their analysis seemed pretty reliable. For sure it was one of the best startups there at Laval Virtual.

PatchXR

PatchXR has been one of the most pleasant surprises of Laval Virtual. I’ve not had the possibility of testing it for a long time because of some technical issues with the Quest, but what I have tried myself and what I have heard from other people that have tried it makes me say that it is one of the most intriguing creative experiences I have tried lately.

I was invited to try PatchXR by Mélodie Mousset, cofounder of PatchXR, who showed me on Twitter some videos about her product. Honestly speaking, I found the videos a bit weird, a bit a mix between a VR manager of music instruments and an acid trip, so I wasn’t very sold on them, but luckily I decided to go anyway to try the experience, because it really surprised me.

PatchXR will be available on various platforms, but at Laval, I tested it on a Quest 2. I entered into the application, and I was asked if I wanted to build an experience or play an experience built by others: Melodie proposed me to play an existing experience, that’s why I clicked on “Build” (I know, I’m a terrible person). In the virtual world, a weird figure that looked like a flower under the effect of weed approached me and taught me the basics of how to build worlds in PatchXR (also speaking as if he was under the effect of weed). One of the basic ideas of PatchXR is that you can build your instruments: every instrument is composed of a physical aspect (e.g. a flower) and a musical ball that has a particular sound (e.g. that sounds stronger, or more like a piano, or more like a bell, etc….). When you put the musical ball into the 3D model of the object, the object becomes a musical instrument, and when you hit it with your controller, as if it was a drum, it emits the sound typical of that ball. Of course, the ball just defines the timbre of the sound, while the amplitude and other parameters are defined by how you hit the object (faster, slower, etc…). The result of all of this is that you can create crazy instruments in VR in a fun way!

There is a video shown to me by PatchXR (the one above) where there are people that have been able to re-create professional musical equipment all inside the experience, so I guess I just tried the easiest of the creation tools, but there are many others that make you create more complex stuff. Anyway, even just the mode I tried was fun, and also artistic, because I could create around me a whole colorful world where everything could be an instrument. It was like that meme of last year where “everything was a cake”, but in this case instead of a cake, everything was an instrument.

But there’s even more! One cool feature is that you can record yourself while you play the instruments… so you create some instruments, then you press a record button and can record your performance. After that, pressing “play” you (and the other people in the room) can enjoy again you playing. And you can do that how many times you want: it means that you can record yourself multiple times playing different instruments, and when you play all these tracks back, you can see the multiple version of yourself playing together like a band!

But Melodie made me go deeper into the rabbit hole, explaining me that not only everything is an instrument, but actually everything is a “patch”, and this means that basically you can create modules that you can rework and combine together to create new experiences. For instance, while we were at a networking dinner, I was concentrating in eating the biggest amount possible of food and she was thinking about PatchXR, and all of a sudden she revealed to me that the tutorial of the experience is not a coded tutorial, but it is itself a patch of PatchXR recorded inside PatchXR.

I was so shocked that I even stopped eating for five seconds. This is the power of PatchXR: it is fully modular, and you can create the instruments you want, the performances you want, and then combine all your creations together to craft a new performance. Most of the whole PatchXR experience may be created inside PatchXR itself!

I also tried exiting the Build mode and going into Play mode. I could see an underwater concert played by three different figures that were playing different instruments and the environment around me was reacting to the rhythm of music. The figures were talking and playing at the same time, and they were clearly three patches recorded before that were combined to produce a new full performance. And it was cool to see also the feature of the environment being sound-reactive, something that was probably also in the Build tools that I hadn’t tried in the tutorial. I admit that the graphical style was not the one I prefer (especially for the style of the weird flower that represents every person), but this is a matter of taste…. anyway the result was very colourful, fun, and interesting. It was a cool performance.

I liked PatchXR because I’ve found it to be an amazing sandbox for creating artistic musical shows for indies. It’s fun, easy to use, and its modularity makes it also very powerful. I’m sure that as soon as it will be out in the stores, music hobbyists will love it!

And that’s it for CleverPoint, PatchXR and Laval Virtual in general! Let me know your impressions about these two experiences I have tried in the comments here below and don’t miss my next article about the Italian event I have helped in organizing!