8th Wall enables augmented reality development for every smartphone… but it has still to improve

Some days ago I stumbled upon a Linkedin post by the XR writer and consultant Charlie Fink, where he told that every AR developer should know a company called 8th Wall because it can become disruptive in the field. 99% of the times, reading posts of this kind, I immediately say “a-ha” and continue scrolling, but this time the opinion came from a very authoritative voice in the field, so I got interested in it. He seemed very confident about this company, so I couldn’t do anything but reading the full article on Forbes. After that, I went to the company website and I discovered that it is already possible to download the 8th Wall SDK and test it… so I immediately registered to be able to review this framework for you. So… is it that good? Mmm… read my article and you’ll know in a while!

First of all… what is 8th Wall?

8th Wall is an American company that is developing an AR framework to enable easy augmented reality development for everyone. So, their mission is two-fold:

• On one side, they want to enable Augmented Reality on every phone, not only the few ones that support ARKit or ARCore (or ARCore through the ARCoreForAll hack);
• On the other side, they want to let developers develop augmented reality applications in a very easy way.

It is very promising, and the fact that it has already raised more than $10M can show that without any doubt.

When I watched the video, I became immediately hyped: markerless AR for every Android phone? Even for my old Samsung Note 4? Wow! I was waiting for the Chinese uSensAR platform, that was announced during the CES and that promised a similar feature, but that product has not been released yet… so having something NOW that could let me, for instance, enable my Portable Love Party app to be used on every device without any marker would be incredible. And also for my consultancies offerings… being able to do AR for every phone could let me offer a better service and get more projects. WOWOWOWOW!

Of course also the fact of an easy development triggered me, even if honestly even working with standard frameworks like Vuforia is quite easy once you know how to do that.

8th Wall, you got my interest.

You had my curiosity, but now you have my attention.

But… does it deliver what it promises? Let me describe it from the point of view of both the user and the developer.

8th Wall from a user’s standpoint

The answer to “Does it deliver what it promises?” from a user’s standpoint gets only one answer:

I downloaded the test app and built it with Unity, then I tested it both with my Samsung Note 4 (running Android 6) and my buddy’s Samsung S6 (running Android 7). Results are highly unsatisfying… not to say worse words.

I launched on my Note 4 the sample app, that should show a table with a TV on it, framing the floor of my office. I was really excited, but when I launched it, nothing showed apart from the actual footage from the camera. Where the hell is the table? Moving my phone I discovered that the table was in a position not visible in the initial frame and had been put next to my chair, in a position floating over the floor.

https://gfycat.com/SpecificAmbitiousBilby

Moving the phone, the AR table started trembling in a sensible way and that is surely a bad sign. I got closer to the table and then I moved away from it and the movement appeared absolutely not smooth. Furthermore, returning to my initial position, the table looked a bit different from before. I decided to go for some harder tests and started moving my phone very fast. The tracking went completely crazy and the table started moving inside my room, changing continuously position and becoming smaller and smaller. In the end, it was like a table for ants.

https://gfycat.com/GrippingPinkHummingbird

The tracking with the S6 was a little stabler, but it was bad as well. I also tried to test the application on my desk, since my floor is a challenging scenario for AR tracking, but things didn’t go much well. With the S6 there is the additional problem that the camera goes in autofocus mode, and during the blurring of the autofocus, the tracking gets completely lost and everything becomes unusable. There are moments when the tracking works in a decent way… and moving slowly the phone, the AR quality is quite good, but an AR framework should work almost always, not sometimes.

https://gfycat.com/InfiniteRegalAdeliepenguin

I have to say that when I tried the AR features of the Motion Still app by Google, that maybe uses an unlocked version of ARCore, I had a far stabler tracking than the one offered by 8th wall. For instance, with 8th wall, putting this alien on my bottle cap as I did in Motion Still seems unfeasible at the moment.

https://gfycat.com/gifs/detail/grotesquetestyjaguar

I also developed a custom app of mine, with a cube in the origin, and when I launched it the cube was shown in a completely random position inside my room (next to my window, at 1.5m above the floor!!) and was also cut in half. What The Fuck.

The user that is inside me says “Nope nope nope”.

8th Wall from an expert’s standpoint

Ok, we have understood that the tracking offered by this framework is not ready for the consumer market, yet. But showing you just the point of view of the general consumer is surely not fair, so I’ll add some lines from the point of view of a user that knows a bit about augmented reality and that has also tried in the past to develop an AR framework himself using OpenCV.

Creating an AR framework is damn hard and requires a lot of skills. Arriving at the point where things “somewhat work sometimes” is quite easy… the hardest part is creating something stable and with “stable” I mean mostly something that:

• don’t lose the tracking easily and that can recover when it is lost;
• when it is performing the tracking, it goes smoothly, so things do not tremble.

Both are very difficult to obtain. In my experiments in developing an AR framework, I managed to show a cube over a surface, but it was super-jerky and it lost the tracking in 30 seconds. I too stumbled upon the problem of the auto-focus and the sad news is that on some platforms it is just impossible to de-activate… and on the ones where you can deactivate it, you get to a point where with a fixed focus the camera always see blurred things.

This just to say that even arriving at what 8th Wall has done is very very hard. Surely there are months of hard work behind that. The test on my floor was very hard to pass and even the fact that the tracking somewhat worked is a good news. They surely need time and money to improve… that’s why they’re raising millions. And don’t forget that they’re a little startup and they’re trying to do the same thing that enormous companies like Apple and Google do.

And I have to add that they’re honest when they say on their website that the tracking is better when it relies on ARCore/ARKit than when it relies on 8th Wall’s proprietary algorithms.

So, IMHO their 6 DOF tracking technology at current stage does not work, but I’m confident that they can improve it over time.

8th Wall from a developer’s standpoint

The answer to “Does it deliver what it promises?” from a developer’s standpoint gets this reaction:

Using 8th Wall with Unity (the only enabled platform at the moment, with native integration to be released in the future) is damn simple. You have just to download the package from 8th Wall website and import it into your Unity projects, add some scripts to some objects in the scene and that’s it. You have 6 DOF augmented reality for every phone for your app. I made an app in 5 minutes, without having any prior knowledge of this framework. That’s impressive.

The true nuisance of developing with 8th Wall is that first of all you have to register as a developer on its website (leaving it your data and your phone number) and then you have to get a special key for every app that you want to develop. If you’ve ever developed with Vuforia, you know the pain… this is something very similar.

Anyway, the docs get you covered very well. I thought about writing a how-to section for the package, but it is almost useless. To learn how to enable 8th Wall in your Unity solution, you can refer to the documentation and to the Unity sample projects.

My advice is to learn how to do an 8th wall app doing those steps:

1. Read the beginning of the documentation to learn how to register as a dev and get a key and how to enable 8th Wall inside your Unity project;
2. Download the package, import it into Unity and enable it with the steps that you’ve learned at step 1;
3. Download the sample projects package using the link above and import the tutorial into your Unity project. The tutorial can be found in the <sample_projects_folder>\projects\8thWallXR-Tutorial;
4. Follow the steps of the Readme.md file of the tutorial. It will teach you the basics of how to transform a standard Unity scene into an AR scene;
5. In case of any problems, refer to the lower part of the docs that have all the overview of the APIs of 8th Wall.

Enabling AR inside your projects just require adding a prefab into your scene (XRController), two components to your Camera (XRCameraController and XRVideoController), one to your light (XRLightController) and one to the root object of your virtual world to be showcased in AR (XRSurfaceController). That’s it. The various scripts allow a great flexibility: for instance, the XRVideoController is there to let you choose if you want to show the camera stream of your phone on the screen or not. XRLightController is useful to set the lighting of the scene accordingly to the real lighting of the real setting where the AR framework should run. You can choose to use or not the various scripts to decide the features that you want to add to your AR experience (for instance you can avoid using XRLightController if you don’t want to use adaptive lighting). I loved this flexibility.

(Just a note if you get stuck during the build stage, if Unity says you that “Gradle build failed”: change build system from Gradle to Internal)

In the doc it is also explained how to add shadows to the virtual objects so that they get projected into the real world… that sounds really cool and it is easy to be done (just requires adding a new material with a particular shader). I’ve tried it and it works.

Apart from this high-level scripts, looking at the APIs I’ve seen that it is also possible to make various interesting calls, for instance, to know what the detected surfaces are, what is the position and rotation of the camera or if the phone is ARKit/ARCore enabled or not. These functions are very useful to add some additional logic to the application: for instance, it is possible to get the detected meshes and color them inside the experience. Or add colliders to them at run-time.

Looking at the APIs, I’ve discovered that the programs developed with the 8th Wall framework rely on ARCore/ARKit if the underlying hardware supports them. So, you can develop with 8th Wall once and then deploy on all phones: if they support ARKit, they’ll use ARKit tracking; if they support ARCore, they’ll use ARCore tracking; otherwise, they’ll use the 8th Wall tracking (the one that I’ve tested above). This means that in any case, your users will be able to use augmented reality and that the currently-buggy 8th Wall tracking will be used only if the phone does not support a better framework. This is very powerful: you use one framework to rule them all… if the engineers working in this startup manage in making their tracking technology better, 8th wall may really allow us developers to develop AR apps for everyone. And in a very easy way. Just wow.

From the developer’s console, it is also possible to see statistics about the usage of the app, seeing for how much time it has been used on the different kinds of phones.

Another cool feature is that the framework includes an automatic update feature: there’s a script that allows you to download the latest version directly from the 8th Walls servers without leaving Unity. That’s very handy.

The only critic that I make at the moment is that the 8th Wall SDK doesn’t allow for fast testing from inside Unity. With HoloLens and Aryzon there are some tools that you can use from inside the Game window of Unity to move the camera, while the 8th Wall doesn’t offer anything similar. That’s bad because this means that every test should be run on a phone, with enormous deploying times.

Final impressions

After having reviewed the SDK, now I understand why people are pouring money into this company: because it offers a service that is very hand for us AR developers. The ability to develop a markerless AR application once and then distribute it being sure that every user with whatever phone will be able to run it is very powerful. The only thing that they need to ensure is that the experience is a good one on every kind of device and to do this they have to improve their tracking technology. I’m skeptical that they’ll be able to reach the standards set by huge companies like Apple and Google even on older phones, but I think that it is possible that they’ll be able to guarantee something usable on every phone. If they manage to get there, they’ve won.

Their only risk is if an engine like Unity starts offering the same abstraction internally: in this case, either they get acquired by Unity or they are condemned to shut down. You may say that they have still the advantage of offering 6DOF tracking on every phone… but the more we’ll go on, the more these models of non-AR-enabled phones will become rare: all the future iPhones will support ARKit and I think that more and more Android phones will support ARCore. If 8th Wall wants to become successful in the future, it has to establish its dominant position in the market now that a lot of AR developers like me needs it.

They have a good idea and potential, for sure. I wish them luck.

UPDATE: The 8th Wall company has replied to this review appreciating it and highlighting how they’re actively working on the tracking technology (“our tracker is still in it’s early stages and you can expect it to get better going forward. We have a top-notch engineering team that is hard at work on this and we expect continual improvements with each release”) and have amazing new features coming down the road. I can’t wait to test them.

And that’s it for this review. If you liked it, I invite you to try 8th Wall by yourself by going to their website. But before that, please share this article with other XR innovators and subscribe to my newsletter to support this magazine!

(Header image by 8th Wall)