Considerations about using HoloLens (or any other AR glass) inside an exhibition

In a previous post I highlighted the problems that current generation of augmented reality headsets have (if you haven’t read it, read it!). Today I want to show you the problems that can happen while showcasing an augmented reality application in an exhibition or such.

On Friday I went to Novara to test a project I’ve been doing with Politechnic of Turin professor Andrea Bottino (and his right wing Francesco Strada) involving the use of HoloLens for medical applications. Purpose of the test was to make doctors to try our solution and make them express feedback. No-one of them had ever tried HoloLens and some of them had even no idea about what HoloLens were… so they were the ideal people to test our super-nerd stuff.

Feedback about our solution has been great, but this is not the topic of this post… (but a good reason to be happy)

Here I want to talk you about problems I had during the test… and how I tried to address them. It has been the first time that I’ve used HoloLens inside a public showcase with non-technical people, so I think it’s interesting to tell you all the problems I had.

Hygiene

Let’s start from the hygiene: how to guarantee the hygiene of the headset between consecutive usages from different people? Well, you can’t.

With virtual reality it is far easier, because now there are covers for headsets: for our exhibitions, we strap a waterproof cover to the internal part of the headset (the one that touches the users’ face) and then we use sanitizing wipes between consecutive usages to guarantee hygiene. Furthermore, we also clean the internal lenses with lens cleaning wipes… this way we can offer a basic hygiene service to all our users (a service that most of people showcasing VR continues to not offer).

With HoloLens, there are not covers on the market yet (there are too few HMDs out there, so I guess covers wouldn’t be profitable)… so this solution can not be employed. Without that, using only sanitizing wipes can be an issue, since these objects release their liquid on the plastic, ruining it: at the end of an exhibition, my VR cover is clearly ruined by this agressive liquid, but I don’t care since it’s only a cover. Ruining the plastic of a $3000 AR toy is another story. Furthermore, the internal part of the internal HoloLens ring that touches the users’ hair is covered by velvet, which is clearly uncleanable. Furthermore is still not clear to me which part of the glass I can touch with a wet cloth without making it to short-circuit.

So, until the moment someone invents a cover for HoloLens, it’s hard to keep them clean. Luckily they have not a foam where they touch the user face and the glass keeps itself a bit more distant than virtual reality headsets, so hygiene is bit less important than with VR headsets.

But I wanted to guarantee at least a minimum of hygiene between groups of people, so I used the lens cleaning wipe to clean the lenses on the inside and outside; and a slightly-wet (wet with simple water) cloth to clean the parts of the hmd that get in contact with the users’ skin: the forehead ring and the nose mount. It has been not a 5-star service, but at least it has been something.

About hygiene, I should also talk about the hygiene of the exhibitor (that is, me): as with virtual reality, I had to put the headset on the head of people and regulate it to make it comfortable. This means touching hair of people… and sometimes they’re a bit, ehm, oily… and I had the instinct to puke instantly. The solution for that: sanitizing wipes for the hands… or maybe offering a free shampoo included with the demo? 😀

Comfort

Guarantee comfort of people wearing this kind of headset for the first time is not easy. Things that I learned:

• The nose mount has to stay on the top of the nose (the part closest to the eyebrows). If you put it even slowly lower, due to the low FOV of the device, people would see the virtual scene on the upper part of their field of view and they will have to constantly look up with their eyes and that’s not comfortable for eyes’ muscles;
• Close firmly the headset or when moving the head some people may make it to move and lose focus of the app;
• Different people have different shapes and sizes of their heads and you have to understand how to make the headset fit with all these shapes… and it ain’t easy. Here the only advice I can give you is to use the internal ring to help you with people with a little head: the smaller the head, the more vertical the internal ring has to stay (Hololens has an external ring and an internal ring: the internal ring is the one that you use to tighten the headset to the head; it can also rotate to make it adapt to the head of the user).
• People with glasses can keep the glasses with the headset, but the headset will never have a perfect fit with their head this way: furthermore, due to the fact the glasses already occupy the position of the nose where the headset has to rest (the one of the first point of this list), usually they have also problems in seeing the augmented scene. So, while they can keep the glasses, it is usually advisable that they remove them.

Visualization

Don’t let the light of the sun to enter the room or the virtual elements will be hard to see. With artificial lights this issue gets far better. Anyway, making the room dark is not a smart solution: first of all because it makes the idea of augmented reality to lose sense (use VR if you don’t want the user to see the world around him!) and then because HoloLens tracking needs some light to work. So, put some curtains at your windows.

Then, be prepared that all people would tell you that FOV is terrible and that the augmented scene is augmented only in a small part of their vision. We all know it, guys… we’re waiting for Redmond guys to solve this issue…

User interactions

FOV is terrible… have I already told you that? This means that if you have put some virtual elements in a position that is not directly seen by the user at the beginning, he will never notice them. Really. Since we required people to look for elements in the scene, we explicitly told them to look for those elements… and even with this suggestion, they had hard times in finding them: often I had to point myself the direction that they had to look in order to find them. This is an interaction killer.

Then, air-tapping is terrible. Air-tap is the “click” for HoloLens: you make it by putting your index finger straight in front of your eyes, then closing it super fast. If you want, you can help with your thumb, starting with the index straight and then making it to pinch super fast, putting its fingertip on thumb’s fingertip. Well, this gesture is completely abstruse for non techie. Most of people had problems in performing it, since it requires you to have:

• index finger perfectly straight at the beginning;
• all other fingers closed;
• hand in front of your eyes, with palm that is parallel to the direction you’re looking;
• (+ to be right-handed, since we noticed that with left-handed people it works slightly worse)

Furthermore, the hardest thing to tell people is that this air-tap works as a click interacting with the object that you’re gazing with the cursor that follows your head (and not with the object pointed by the index). All this makes people’s head to explode, trust me. Lots of people opened all fingers, most of them did not keep the hand parallel to face, some of them had banana-fingers so HoloLens could not detect them.

So, from a computer-vision point of view I understand why Microsoft chose air-tap as an interaction… but from an usability point of view I can say it is a pain in the a**. I myself needed some practice to start mastering it.

And I won’t tell you all the problems we had when using Vuforia markers… people putting their hands on the markers or looking up while the marker was down and expecting the AR to work…

A last notice on vocal commands: we completely removed them, because the system understood only commands pronounced in English with a Redmond accent (and we’re in Italy, far away from Redmond). In our internal tests, for example, we discovered that system never understood the keyword “Charge”… to make it understood the command, we had to prounciate the word “George”. We were in the lab screaming “Geooorge”, “Geeooorge” and seemed like three crazy George Clooney groupies. So, to make the system usable, we removed all vocal commands.

Guidance

Since we noticed from internal tests that people had difficulties in understanding how to interact with HoloLens, we applied two strategies:

• KISS: we kept the interactions simple and stupid. Most interactions were performed only with tap… so it was necessary only to learn the tap-gesture;
• Super guidance: the program began with a training phase where we taught them how to interact with HoloLens (GIF + explanation of the air-tap gesture, etc…) and then in each phase of the program there was a voice telling them what to do (this was the voice telling them for instance “look around and search another person that can help you”, so they looked for other objects in the room).

These two approaches helped a lot… but remember an essential law of exhibitions: “PEOPLE DO NOT READ, DO NOT LISTEN”. After the tutorial and and explanation made by voice with me showing the gesture, they continued to do the air-tap the wrong way. So in the end I took the hands of people with violence and closed the other three fingers and opened the index to make them understand. I know, it may seem only an excuse to sexual harass beautiful women, but it worked 😀

When some people continued to fail in performing the tap gesture even after the harassment (banana-finger issue, I think), I did a small trick: I put myself behind the user, with my hand in front of the hmd of the user and told them “look, you have to do this way” and performed the tap gestures while he/she tried to emulate me. They thought they learned the gesture, but instead it was me tapping for them (HoloLens do not detect very well which is your hand and which one is not). It’s cheating and deceiving, I know, but when you have only 5 minutes for each demo… there’s no time to be mister-nice-guy ;-).

Then there’s the other problem of knowing what your users are doing. HoloLens have a mirroring function towards the PC, but:

• It has a delay of 2-5 seconds wrt what the user is doing;
• It makes FPS worse;
• It doesn’t work for Unity apps (you can only see the virtual elements, not all the augmented reality stream);
• It drains the battery faster.

Using a Unity app and having battery problems (see next paragraph), this approach was not usable. So, how could we guide users? First of all our app was very simple and we knew it by heart, so was very easy for us to understand what was the current phase and which problems the user could have. Then we put a lot of audio-feedbacks and 360-audio emitted by HoloLens is clearly distinguishable if you’re close enough to the hmd, so we stood next to the user and tried to understand from the sounds what he was doing. Then, dialogating with users is important, so we were costantly asking them what they were seeing. As a last resort, we removed the headset from the user and looked with our eyes what was the issue.

Battery

Battery is a serious issue if you want to make a long test with HoloLens. Our system consumed something like 4% of battery charge for each user (10 minutes of usage circa), so after 25 users the battery would have died. We tried to mitigate the issue by adding some little pause moments when to charge the device, but it didn’t suffice. Luckily a guy there had a big battery-pack, so we have been able to make the users try the device while they were attached to the power-bank… not that professional, but it worked.

Here the takeaway is: make an estimate of how many users you can handle with HoloLens battery and calculate all timing accordingly; furthermore, TAKE A BIG POWERBANK WITH YOU.

Stability

HoloLens is still unstable and it froze like 2-3 times… plus other 7-8 times it quitted automatically from the app. This problems got worse when battery levels went lower. Don’t calculate your time thinking that everything will be alright… malfunctioning are quite normal when you use an experimental technology, so consider them in your timing table and smile to your users telling them that they’re normal.

And that’s it. This is what I’ve learned in this first “exhibition” with HoloLens. Hope that this will help any of you that wants to make a public demo with this interesting device. If you think that this can help other people too… please share it!