Eido Inoue tells us everything we need to know about HADO and location based augmented reality

When I interviewed Kevin Williams, an expert in the LBVR market, he said that there are also some experiments with location-based augmented reality and he mentioned a Japanese arcade AR game called HADO. For sure you all know HADO: even if you don’t remember the name, you have for sure seen one of those viral video where people with an AR headset on the head throw fireballs each other.

Well, after that article, I discovered that the CTO of MeLeap, the company behind HADO, is one of my readers! :O He contacted me and of course I asked him to have an interview with me regarding his job in LBAR. Since I was very curious about it, I asked him a lot of questions… and being him very kind (he’s Japanese, after all), he answered them all, so the interview is quite long, but I promise you that it is incredibly interesting. Go on reading, I bet you will enjoy this journey into the present and the future of location-based augmented reality entertainment.

Hello Eido, introduce yourself to my readers!

My name is “Eido Inoue” (in Japanese: 井上エイド), and I’m a Japanese national, born in Washington D.C., who first immigrated to Japan over 25 years ago. I’ve been married to an Osakan for over twenty years, and I have a Japanese-American teenage daughter, of which the only thing we have in common is 3D computer illustration and animation. Her graphics workstation is far more impressive than my laptop in my home office. 💸

After majoring in computer science at a U.S. university, I started as a software engineer for a small Internet Service Provider in Osaka City in the mid-nineties. From there, I worked my way up through various companies to software management, then director, and since last year I’ve been the head of all things technical at meleap inc. in Tokyo.

I the beginning I worked in enterprise software, working on Linux and open source software with Red Hat for a total of over ten years. Between that, I worked on consumer cloud software with Google for five years… managing engineering teams for Gmail, Japanese mobile, Google Labs, and social (Blogger etc.) teams for Japan. Emoji and the poo emoji in your messages is partially my fault. 💩 My first hands-on programming exposure to VR and AR came recently through being a director at a French/Japanese mobile game company and creating AR prototype interactive toys (mostly with AR.js, ARToolkit, and Vuforia) for business cards to promote company logos and brands.

In my spare time, I enjoy training and running full marathons around the world and playing games on my PSVR. I’m currently hooked on AstroBot and Moss, having just finished Déraciné.

What is Meleap? And can you describe me your product HADO?

Meleap (pronounced “may-leap”) is a combination of the words “Merry” and “Leap”. As of 2019, we are an expanding private company with about 40 people based in Japan and funded by Japanese venture capitalists. We also have a small office and a couple of people overseas in Europe and America. We celebrated our 5th anniversary as an incorporated company last month. We specialize in a unique genre we call “technosports”, which is basically augmenting (as in the “A” in “AR”) real (as in the “R” in “AR”’s reality) sports: be it individual sports, team sports, or even motorsports. We make things for both the players and spectators.

We try not to equate what we do to “e-Sports” (which we respect but don’t compete with), which is more the traditional paradigm of video gaming: hand-eye coordination, where the only things in motion are perhaps your fingers or wrists and you stare at a monitor mostly motionless.

With “technosports”, you will sweat. And you will be out of breath at the end of the game.

“HADO” is a name we use either standalone to refer to our flagship and most popular team sports game, but it also refers to a genre of games where an energy ball is released by the player through gestures.

The word “HADO” comes from the wave attack in “Street Fighter” done by Ryu and Ken, where they put the palms of their hands together and push a ball outwards towards their opponent, saying “HADŌken!” (波動拳), which translated character by character from Japanese means “wave”-”motion”-”fist”.

In tournament versions of HADO, two teams (red and blue) of 3 players are in an indoor court and must stay in bounds and offsides. While dodging energy balls from opponents (which involves high speed running, lateral shifting ducking, and leaping around the court), a player can either raise a (limited amount) of shield / barriers / force-fields, charge for release, or release an energy ball at an opponent targeted via their HMD/HUD. Both the shields/barriers and players have a concept of health/hitpoints, and knocking a player out earns your team a point.

The team with the most points after 80 seconds wins. If it is a tie at the end of 80 seconds, sudden death overtime occurs.

Each player has four characteristics that they can individually tune prior to starting: speed of the balls, barrier strength, strength of energy balls, and charging speed.

It takes both agility and teamwork (working with other teammates on how, when, and where to deploy the limited virtual shields) to win.

If you do not move or simply walk slowly around the arena, you will lose against any professional team, who are all in good physical shape. On the other hand, speed and agility alone will not win either. Slightly slower teams have been known to beat faster teams by using better teamwork and strategy.

What are the software and hardware platform used by HADO?

Our software and hardware engineering and research teams have used a variety of equipment that rides on top of our in-house developed technosports platform, ranging from the HTC Vive Pro and its motion detectors, the (now unfortunately defunct Myo) sensors, the ZED stereo cameras, and Microsoft’s HoloLens. Software-wise, we use Unity, Vuforia, and OpenCV. However, we try to develop our platform to abstract the software and hardware because in this business, the hardware and software platforms are evolving so fast (and with that, old technology either quickly becoming obsolete or going out of business) we have to be nimble and adapt quickly as the XR world evolves.

We have very unique needs for our products that makes most off-the-shelf XR solutions not viable for us:

• The hardware platform has to be lightweight, completely portable, and robust enough to survive rough semi-contact sports. This rules out 99% of most out-of-the-box consumer, enterprise, and development XR solutions: they are heavy, fragile, and expensive. It’s not uncommon for HADO players to fall, collide with each other, or run into walls at sprinting speeds in excess of 20km/h, which would quickly destroy a HoloLens or Vive Pro costing thousands of dollars. Don’t get me wrong, they’re great technologies, but they weren’t designed to do what we do;
• The software platform has to be robust enough so that it provides a true competitive sports experience. That means it can’t be susceptible to common XR problems like “drift” (6DoF XYZ/yaw/pitch/roll misalignment due to harsh, fast, sudden, and extreme movements of the head/body) nor can it even lose its tracking or need recalibration during the middle of an active tournament. The tournaments are played for money, and the professional teams invest their own time and money into their training, practice, and travel, so they can’t be let down by the technology when everything is on the line.

Currently, we use custom-made HMDs and arm gesture sensors that are designed to:

1. Be very lightweight — ideally lighter and more comfortable than a protective helmet you would use in baseball, rugby, or cricket;
2. Be hygienic (they don’t absorb sweat or odor and use hypo-allergenic plastic and rubber),
3. Both provide a VR/MR like completely encompassing experience while intentionally opening up the sides of the helmet for peripheral vision with the real world. We have found that this reduces the chance of “VR motion sickness” as well as physiologically allowing players to maneuver at full speed with confidence.

The HMDs for 2018 actually used iPhones† for both the gesture detection and HMDs (that communicate with a local detached server via Wi-Fi and Bluetooth), but we have used Android, Vive, and Myo hardware in the past, and we continue to evolve the platform as the state-of-the-art pre-made AR/MR gets closer to something that can be used in aggressive arena play.

† A notable exception to this is our “KART” (go-kart) series of HADO products, which use the HoloLens due to psychological fears associated with wearing a non-smartglasses (WaveGuide lens based) HMD while driving. There are technical reasons too: when people “drive”, they look down at an angle, and AR doesn’t track either expansive markerless planes (edges beyond the FoV) or markers viewed at extreme angles (>50°) well.

When I saw the videos about HADO, I thought it was just a concept, since they are all in CGI… so do you confirm that this is a real product that people can try in Japan? And have you some gameplay shooting of real people playing with it?

I’m not sure whether to blame or compliment our marketing department for that viral video on social media 😁. I assure you if you type “HADO” into YouTube rather than just watch the ads in your social media feed, you’ll find our official channel as well as hundreds of videos of real gameplay… including originally live-stream tournaments (complete with J-POP guest stars, announcers, play-by-play and color commentators, dance squads, and audiences) for our spring, summer, and end-of-year final world cups.

HADO is currently in 23 countries at 52 LBE arenas, and about 1.3 million people played it last year according to our server stats. By players-and-plays, it’s actually more popular outside of Japan (60% of our business) than inside it… especially in southeast Asia (Thailand, Korea, Malaysia, and Singapore). In Europe, there are venues in Britain, Spain, France, Serbia (by raw play stats it’s the most popular area in the EU; don’t ask me why), and Russia.

Spectators watching live don’t watch through an HMD or phone; there are giant projection monitors above the arena as well as monitors throughout the arena where they can see the virtual energy balls and shields and overlays for player stats and scores. What you see on YouTube is what the spectators watching the action live to see through the stats screens.

Additionally, most LBEs use the same (what we call HADO “EYE”) technology in their locations so spectators can see what’s going on, and so that the players can watch themselves after the game (replays) to evaluate their own performance.

Here’s a video, cued up to my favorite part, of the two top teams in Japan, playing for a ¥2,000,000 grand prize (>€16,000), in the final round of our 2018 World Cup: (go to the 5 hour mark, if it does not start automatically there)

This is what HADO looks like from the perspective of our “EYE” (taken inside our volume at our development office, which is actually located underneath Tokyo Tower)

How has the idea of HADO born?

The founder and CEO of our company, Hiroshi Fukuda, was a big fan of the manga and animé “Dragonball” and its main protagonist “Goku” (who you’ve probably seen people dress up as for cosplay or Halloween due to his distinctive large spiky hair and orange jumpsuit). Goku and a few other characters in that series had a signature “move”, the “Kamehameha” attack, which, just like the fighting video games, involved shooting an energy ball/wave from one’s hands.

The company had been dabbling in AR/VR/MR projects for some time, and the CEO/founder asked his engineering team if they could whip up a prototype that allowed him to fulfill a childhood dream of being his favorite character.

We did it… and eventually, we got the idea that we could use it as a basis for creating a new type of sports.

What is the feature that people love the most about it?

Similar to how there are tourists that drive around Tokyo in Go Karts dressed as Mario characters (much to the consternation of Nintendo), I think everybody at one time has always dreamed of what it would be like to have a superpower, or live in a fantasy superpower filled world, similar to ones they saw on television or comics as a kid.

The first time you throw a fireball from your hand, you get a rush.

The next time when you throw a fireball at another real person and it hits them, then you really get a thrill.

I think that’s a universal that we have in common with VR; because the experience is immersive, it makes you believe you really have transformed not into a different world (that is better than reality), but you yourself have transformed into a better version of yourself.

As for spectators in the pro tournaments, they seem to get the same excitement from watching the competition that you get from watching a traditional non-augmented sport. The thrill of victory and the agony of defeat for your favorite team. I’ve heard audiences gasp and cry when their team gets to overtime in the final do-or-die rounds of the World Cup. They don’t seem to notice that the balls that the players are throwing and the barriers they are erecting aren’t really real. They get immersed in the augmented reality even though they aren’t using a phone or headset.

What have been the main difficulties in creating this AR experience?

At Red Hat, the objective was to keep the operating system platform stable — API and ABI wise — for as long as possible: years to even a decade. This was to please its conservative customer base, which tended to be financial and other big Fortune 500 customers that don’t like change (because it’s expensive); they often have tens of thousands or more machines to administer.

At Google, the change was rapid but much easier to control, as everything from the software to even the servers were designed in-house. Thus Google controlled almost the entire stack that the client web software used — right down to the hardware in the data centers.

Although VR and AR have been around in the research labs since the 90s (I actually used SGI’s headset as an undergraduate working in the college labs), it really hasn’t taken off as a commercial product until recently.

Because of this, compared to the change at my previous companies, the evolution of AR and VR is not just exciting and optimistic, but it’s “disruptive” to the point of being also described as volatile. Brand new hardware and software and advances are happening all the time, making yesterday’s way of doing things obsolete before the sun rises. And with that technology obsolescence also comes companies and products that disappear right after you invest and adopt into their solutions.

Thus, I’ve had to use my experience as a software engineer to focus on the platform and abstraction of the different layers of the stack, so that when a layer at the bottom (such as the Myo) middle disappears due to a supply chain problem, we don’t have to re-write our apps to work with newer, but fundamentally incompatible technologies.

Doing this, while maintaining performance and flexibility, is quite a challenge. Although it’s an exciting one.

Doing LBAR (location-based AR) is quite unusual at the moment. Why have you decided to bet on it?

The first reason is we provide a sports product similar to other sports with defined open playing fields. Many people who live in cities are probably familiar with futsal courts on rooftops in cities. Or tennis or squash courts. If you’re an urban Japanese, you’re familiar with baseball batting cages or golf drive and putting practice ranges. You can’t really play HADO in your bedroom or even living room; our courts can’t really be smaller than 6m × 10m. And even if you could, it requires specialized custom hardware that you can’t purchase.

The other reason is technical. The red and blue “walls” on opposing sides of the playing field (the 2018 designs resemble street wall graffiti) are actually gigantic tuned AR positioning markers. We have found that even the most advanced markerless software available today in 2019 (the latest ARCore, ARKit 2, Vuforia, 8th Wall, and even Niantic’s “Neon” demo, which I’ve tried) really can’t deliver a robust, driftless experience for 80 seconds when players are moving at full athletic speed. Not even walking for 80 seconds. Because HADO teams are always facing each other and aiming at each other, they are almost always facing an AR marker, so the HMDs can maintain precision and tracking because they’re constantly calibrating seamlessly in the background.

How is different doing LBAR than doing LBVR? As platforms for LBAR is conceptually better Hololens or Magic Leap? Or is it better a passthrough VR headset used for AR/MR?

Pedantically speaking, some may refer to what we do as MR (mixed reality) rather than pure AR, as the HMD, unlike what a HoloLens or Magic Leap waveguides can do, can completely replace your entire field of view with the exception of the extreme sides of your peripheral vision.

One thing we have noticed is that most people are unwilling to move very quickly when wearing a proper VR helmet. And when in a moving vehicle, like the go-karts used in our “HADO Kart” series of games, even our open-air HMD “AR” is too scary for that speed. It is for that reason we use Microsoft’s HoloLens for our KART games.

How do you envision the future of LBAR?

I think that any traditional sports based LBE (such as laser tag, or paintball) will greatly benefit from technosports. We use LBEs for technosports because HADO requires true freedom of movement in a real defined and controlled environment (a sports playing field).

As long as people are willing to engage in activities that benefit from human movement (ex. sports), these activities can be enhanced with technology. Including AR and MR.

From a business perspective, I think that technosports will broaden its appear from not just participants, but to audiences as well.

However, in order for any LBE (be it LBAR, LBVR, or something completely unrelated to XR, such as the old video game arcade center) to succeed, in addition to all the standard business model checkboxes (price, location, etc) you have to have a business model that, at the minimum, checks three boxes:

1. The experience can’t eventually be replicated (or a close approximation) in your home (or portably in your hand wherever you go) for an affordable price;
2. The experience has to be thorough enough so that people will want to take enough time to want to invest a significant amount of a day in it (such as planning an evening with your friends). This includes providing enough depth and diversity of your experiences;
3. The experience has to be rich and complicated enough to encourage repeated use of the experience. People who experience it should feel that the more time and play they invest in it, the better the reward will be for them, and that reward should be compelling enough to demand the time and money investment from the customer.

Much of the early LBVR (which are now defunct) based their business model almost exclusively on the premise that VR/MR technology is too expensive for the masses, or it’s too difficult or bulky or time-consuming to set up in your home. They used gimmick attractions (usually glorified 3D movies with some licensed IP [intellectual property] famous characters) whose replay value was dubious. Even if they kept their content fresh (most didn’t), just being an immersed popular character in a 3D scene with 6DoF isn’t usually enough to encourage repeated usage in my opinion.

As both XR (VR, AR, and MR) progress, the technology will become more and more affordable and powerful and accessible to the average consumer, and the LBEs whose business models over rely on customers not being able to experience it at home or school or work will have an increasingly difficult challenge staying viable.

AR is an even more difficult business case for an LBE, because 99% of AR technology is designed to be portable and cheap. Who hasn’t at one time installed an AR game on their phone? AR is now standard equipment, built into the ROM, of Google Pixel handsets with their Camera’s “Playground” functionality. I was even using AR with the old PlayStation Vita.

In conclusion, in order for “LBAR” to make sense, you need an AR technology that needs a specific good real (business and/or technical) reason to need a fixed location.

What is the biggest lesson that you have learned in these months that you are working in this sector? Some suggestions for who wants to enter the field?

We interview engineers constantly, and we regretfully have to reject over 90% of the resumes we get, even though they often manage to “check”, at a superficial level, many of the requirements in our job descriptions for game engineering, front/client-side and back/server-side software engineers.

If you’re coming from a gaming work background, it’s important that your experience is with 3D. Although we use Unity in-house for development, we don’t require it specifically. If you are a master of Epic’s Unreal, Amazon’s Lumberyard aka CryEngine, or even a proprietary 3D in-house game engine (we have a lot of ex-Sega engineers), you can probably transition to a new engine if your software engineering fundamentals, and understanding of 3D graphics and the problems relating to performance, are solid and sound. Having extensive 2D or web / social gaming development experience, even with technology we use, isn’t something that translates very easily to the world of 3D.

Finally, in addition to a basic understanding of (video) game and sports theory (ex. what makes a game fun), a personal experience with the hardware of XR is important because the UX for a XR based experience is more critical compared to a traditional app or game — as the experience is literally enveloping you as opposed to just on a screen and at your fingertips.

What are your future plans? And what are the ones of your company?

Last year meleap successful acquired “Series B” funding from about a half dozen investors, after having met the criteria for getting Series A and startup funding in the past. So far we’ve raised well over the equivalent of about $11 million.

e-Sports was a demonstration sport at the Asian Games last summer; meaning countries competed against each other except the medals didn’t count. e-Sports will be a medal sport at the next Asian Games. e-Sports is also planned to be a demonstration sport for the Paris Olympics in 2024.

We believe that if e-Sports can become an internationally recognized sports, then so can its more physical and “traditional sports-like” cousin: techno-sports.

We plan on continuing to expand the influence of both our technology, franchising it to more world distributors and more LBEs in more countries, as well as increasing the number of sports, the number of tournaments (we currently host over 80 a year), the number and reach of broadcasts, and number of pro teams that play technosports. Additionally, we’d like to focus on providing AR/MR experiences for spectators and fans of sports as well, and perhaps allow them a more interactive way to participate (similar to what Formula-E does).

Additionally, we are expanding our business model by branching out into streaming broadcasting over the internet, and basically live broadcasting sports have been using AR techniques (albeit analog until recently) long before it became commonplace on smartphones.

Anything else that you want to add?

In my bookmark folder for “AR/VR/MR/XR” news, I only have about a two dozen sites for getting commercial and technical news about the XR field. Thus I appreciate the work your site does for keeping me and others informed as there’s not enough original and timely tech info on the subject, despite its rising popularity.

Keep up the good work!

I’m really flattered by Eido’s words and I really want to thank him enormously for this interview, that is one from which I have learned the most in these two years of blogging.

I encourage you to visit Meleap website and to try HADO if you ever go to Japan, because they are really doing something innovative. And in the meanwhile, please subscribe to my newsletter and share this post!