KLM’s Chris Koomen teaches us how to use AR/VR in the civil aviation sector

Hello Chris, please introduce yourself to my readers!

Hi, I am Chris Koomen, a Virtual Reality specialist and consult at KLM and board of advisory members at XR Bootcamp. I started my career at KLM 13 years ago as a ground mechanic for wide-body airplanes and playing around with VR starting from 1996. In 2015 my girlfriend came with a Cardboard and with no experience with Unity I created my first 360 photo which I’ve shown to my manger. This was such a game-changer that she asked me what we could do with that. I figured out that if we played the sequence of the photos correctly we could make something like a movie. There started to grow the use of interactions in 360 videos. Because my knowledge of Unity was (and still is) limited, I requested the help of external suppliers with whom we’ve built the very first 360 interactive videos with CGI in it: smoke and fire in a real aircraft and an elevator filled with smoke.

It took off from there very fast: I went to another department and started the VR team there, first with an intern, then a developer, and within 6 months we had a team of 8. Most of the time it started with an idea and when that succeeded we went to the business department and see what they wanted. If they saw the potentials of it, we started to define what exactly the need was to then produce the tools they needed in a POC base…. a bit big POC 😉 … in the end, it turned out most of the time in a fully functional training part. 

Nowadays we have a lot of training solutions, from 360 videos with a flight in an airplane, or scenarios about evacuation or operating airplane systems in maintenance, to fully CDI courses where you can connect the Jetway bridge, do a pushback in all kind of weather, familiarize in cockpits and even start the airplane.

We all know what KLM is, but… why has an airline company an XR department? And what you do there?

I have to split this answer into two. We have KLM digital studio, where we made the proof of concepts for the company. This is where we did all the experiments to find a way to work in a more cost-efficient way while also developing environmentally friendly tools to reduce the carbon footprint. This can include tools for the ground crew to communicate, tools to make access to rosters easier but also AR and VR. The last one was my part and I have only focused on that. A colleague of mine is working only on AR with ARKit and is building an app for maintenance. From there, if the POC is a success, the next (IT) department will take it over and industrialize it. So the roster tool will go to HR and the VR one to the XR Center of Excellence, that is part of the IT. It’s like creating your startup within a company. An Innovation center.

We were exploring the possibilities and limitations of VR on a POC level. We were the first ones using pass through on the HTC Vive, using a portal to get in your scenario, we were the first ones with hand tracking on VR headsets where you could flip switches, turn knobs, and push buttons in a VR world overlay. And we had the first multi-player VR where you could grab objects and give them to others without being physically close to each other. The team is legendary now.

Now I am working for the XR Center of Excellence, and our goal is to industrialize AR an VR in the company and do a little on the experiment side. As you can imagine, there is a lot to take care of once you deploy something on the enterprise level. Just as when you develop an app for your company, you need to be prepared for a lot of things. Now VR is something new so we need to “find” a new way of thinking and overcome new challenges. For example, creating and processing 360 videos creates a lot of data. Not all enterprises are happy when you ask 8 terabytes of storage room, so you need to solve that; then you have the internet connection, security, and how you control the devices remotely; where do you store your data, how do you collect and secure data and what you can do with that… tons of new questions to find the answers for.

There are a lot of people fantasizing about using VR to entertain themselves during international flights… do you think it will ever be a possible scenario or the logistics (e.g. guaranteeing the hygiene) would be too complicated?

In 2018 we did a case study where people were invited to use virtual reality in a BYOD (Bring Your Own Device) fashion. We chose this method because this way the airline does not have any responsibility for hygiene or content on the device but also no need to certify the hardware on board of the airplane. The IMCA (Immersive Cabin) project aims to evaluate and demonstrate the use of immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) in relation to exponential technologies within an aircraft cabin environment, to enable them to be used by passengers and cabin crew during revenue flight operation.

A total of 43 persons (39 passengers and 4 cabin crew) participated in the tests in KLM’s motion base cabin simulator. Participants used a wayfinding app on two VR headsets (Oculus Go with a hand controller and Samsung Gear VR), as well as on a smartphone which was the reference condition. No headphones were used. Each participant made the same three simulated flights: (1) a normal flight with minor turbulence, (2) a normal flight with moderate turbulence, and (3) a flight with an emergency (rapid decompression, oxygen masks). Participants filled in various questionnaires. Video recordings were also made and analyzed.

Safety is one of the key drivers of any decision within the airline industry. Several safety issues from the airline perspective were identified during the experiment. The main issue identified in the experiment was that some participants in the experiment did not seem aware of the emergency situation or did not react to the requests/demands of the cabin crew. If this behavior replicates in real life, it will be very challenging for the crew to attend to larger groups of VR-using passengers within the time available (between 20 seconds – 1 minute, depending on the severity of the decompression). This may need to be addressed in training or procedures. Particular circumstances can also occur as a result of on-board VR use; one passenger mistakenly interpreted the announcement “keep [oxygen] masks on” as an instruction to keep the VR headset on, and subsequently did so. Announcements may need to be adjusted to take account of VR users and the potential to misinterpret messages.

For this entertainment case, do you think it is possible to use 6DOF devices, or 3DOF would be the key?

Now, 2 years further, we still don’t use VR yet on board but when we can, there will be a possibility to set the HMD to 3dof and play movies. When using 6dof, imagine the passenger behind you playing a boxing game… it is a great massage for the first 30 mins but after a night flight of 9 hours, you’ll probably knock out..;) 

There is also a lot of talking about using VR for traveling at home. I still think that a physical visit can’t be substituted by a VR trip now, but what is your stance in this sense? And have you ever made experiments on this?

A while ago there came a colleague to me with the question if I could help her with a challenge. She had a best friend who’s ill and won’t make the end of the year. This person had just one wish, he wanted to see the Northern Light with his family. He already tried to project it on the ceiling and watch it in the living room but still missed the feeling of being there. So they came with the idea of using virtual reality and they needed 6 glasses. I went to them and supplied the glasses and the software. After watching the Northern Light for about 20 minutes, he took off the glasses and said, “I’ve seen the Northern Light, and it felt like I was there”. Now creating an experience like that is not as easy as it sounds. It’s not like putting a headset on and you’re there. Factors as environment, sound, and content are all just a small part of it. You need to understand the technology and how you react to your sensory input. For him and his family, this was an unbelievable experience, and that was the goal. KLM will probably investigate the possibilities regarding VR travel but never as a replacement to real traveling.

But it’s not only VR traveling but also less traveling with the use of VR. We are working on a case study where we use GLUE to see how we can use VR to reduce trave:, the reason for this is that COVID-19 forced us to work from home but we have the need of being together. This is what we did for years and is such a standard that we cannot change overnight. Using virtual reality gives you the feeling of being together while at home. VR can be very powerful for illusions. Your brain will quickly adapt to the new situation and will have difficulties splitting reality from virtual reality. This is based on expectations in the past. To give you an example, we did a test with pilots four years ago: we filmed a lightning strike in a full-motion simulator where pilots need to do their exercise, called Startle and surprise. About 50% was reacting to their environment as in real life, like asking questions to their colleague pilot next to them. He of course was a prerecord and did not reply. And after a few months, the first reports came in where pilots described this situation as they had already experienced it before. This implies the brains thinks it has been there before. It couldn’t divide the difference between real and reality: in the end, your brain registers an event recorded from the eye and ear input so it must be real. 

I’ve seen this behavior before when we worked on the evacuation training. There was a situation where you were locked up in an elevator filled with smoke (CGI) and the actors choke in there and fell on the ground. Most people reacted emotionally to that particular situation. They had red spots in their neck, faster breathing, and rapid movements with their body. The interesting part was that if you asked them if they felt something at that moment, everyone said no but their body language said something else. This also happens with CDI where we have fire safety training. The moment something happens you see people change from “having fun” in VR to “surviving” mode and start to react professionally to it.

Speaking about more practical stuff, I bet that your biggest use for AR and VR is for maintenance and training, right? Have you some projects to share in this sense?

We are working on the Embraer cockpit ground procedures right now. this is a training experience for new pilots to familiarize themselves with the airplane and start practice procedures in it. They can power up the AC, start the Auxiliary power unit, start the engines, and put them all off also. This can be done as a captain or first officer, both have their role on board. 

Because we don’t have the developer capability in house anymore (COVID-19 forced us to say goodbye to some good people from the team) we’ve outsourced it and partnered up with Capgemini. There is our development capacity now. 

What other projects have you worked on?

About 90% of all the VR projects are made within AF/KLM:

• The Matterport walk around was to see if we could train people before even getting on board of the airplane;
• The Evacuation training was my first project where we did a 360 interactive movie on the evacuation of the hangar;
• The startle and surprise project
• The IMCA project
• The Jetway bridge project
• The Cockpit training project.
• We also have a Sensiks pod in our lounge on Schiphol airport for costumers.
• …and many more..

What instead do you think could be used to improve airports? And is there some XR projects working in this sense? For instance, I always get mad to find the fountain of drinkable water… maybe some navigation tool would be nice…

Well, Schiphol Airport (Schipol is the airport of Amsterdam, and it is the main airport for KLM) is changing every day, the Monday you have a Coffee bar on your left, and a day later it’s relocated to you right. It’s almost frustrating. I’m often at the airport and never can find a fountain unless they broke a water line somewhere. So that’s just too difficult I guess. Too many changes in a short time.

But, let’s take a pilot, who’s flying not so often to AMS, why not using AR to guide him on the ground? Visualize taxiways, other planes, and vehicles. Give him the visuals on a foggy morning if it is like a sunny afternoon. Even with IoT you can combine that and have all the ground traffic visible. That would make it a lot safer.

Summarizing all of this, what makes AR and VR are so useful for the airline sector?

One of the biggest advantages is that we can do things we’d never do before. We can make things that were always hidden become visible. Train with a fire on board of a real aircraft? You can! See true panels with AR so you can easily access the right compartment? You can! Find your suitcase by just pointing your phone and follow it across the airport?..;) why, because it is possible..;)

What headsets are the most useful for your job, and why?

I really get this question a lot and it’s always the same answer, it’s so depending on your use case. For 360 video we use Oculus Go, because a Vive Index would be an overkill. On the other hand, we also have the pushback training on the Vive index for the visuals and accuracy. For the Embraer pilot training, we use the enterprise Quest for its flexibility, so they can train whenever, wherever they want. Just make sure you pick the right one for the use case. 

How do you imagine the Apple Glasses? And do you think they could be useful also for enterprise people?

I have truly no idea… In the end, it can be helpful in many ways. Of course, I can give you some examples but I can’t give all my secrets away..;)

I’m sure the COVID has impacted a lot KLM, and my thoughts go to all your colleagues and your families. What has changed in your working life since the outbreak? Do you think that VR could help KLM somehow in this difficult moment?

Well, I’ve spent more hours at home in the last few months than I did in the 3 years before at work so… It changed a lot. I have a son born in the middle of the corona crisis, a week later a close friend of mine died because of corona. Those were difficult times: you’re struggling with feelings, on one side you’re happy because our son was born, but we couldn’t show it to someone; then a friend died, but you can’t really say goodbye there; then the work, you can’t go to the office, so we needed to relocate everything to our houses. Suddenly you have a room dedicated to VR at home, a powerful pc, headset, and a crying baby around.. guess where I was..;) we do a lot of meetings in VR now, in Glue but also on golf parkour: who says it needs to be boring space to talk if you can do it in a place you like? And winning from your boss is always fun…

We are now in the second phase of the trial and we see people understanding the benefits of it. Less travel, more effective meetings, and fun! 

What are your future plans for your work?

Our main goal for us is to implement VR in the company, which includes everything from designing VR applications to remote device management. The first will be the Embraer training, where we use all kinds of VR. 360 video, Matterport tools, and CGI, all are in the new KLM Cityhopper initial pilot training starting from November. We are the first in the world having these kinds of training tools for airline crew in their curriculum.

A personal question: why don’t you serve onboard that delicious mango juice that has Emirates? 😀

Because we’ve got Heineken..;) but I have to say with that, Alcohol and flying aren’t a good combination.

Tell me the biggest lessons that you’ve learned in these years and that could help the people wanting to do your job.

I continue learning in this field and learn so much every day. But to point out some things…

1. Start small, with VR you need to show people first to convince them. That does mean you don’t have to create entire training experiences, you might want to test first with 360 videos.
2. Make sure you know the business and they know you. Have clear what you can expect from each other so there will be no disappointments at some point. For example, when you make a POC, make sure you both understand it will be just one part of the training with one tool, one task, one scenario, and so on. So if that’s a success, they can reserve your money for it. Saves you big time while a huge case can be very clear at the same time.
3. Work across the company and make sure you all work for the greater goal, not for your own benefits.
4. Have enthusiastic people in your team. Those are willing to invest in the project.
5. Always, and then I mean really always have fun. We have in almost every tool that we made an easter egg, just for the sake of fun…

Anything else to add to this interview?

From November, KLM Cityhopper will start with its new Embraer training course, and this training has VR a structural part of the course. With this, we are the very first airline that has virtual reality embedded in training. Which of course is a new step in our journey to digitize and reduce the carbon footprint. We got so far because of the extraordinary teamwork across the company with different departments. I can’t close this interview with thanking them all for this great achievement!