How to create a seamless mobile AR experience using AR.js

Today I’m going to write a detailed tutorial about AR.js, telling you what it is and how it can be useful to create seamless simple mobile AR experiences. I spent a lot of time experimenting for this article, so I really hope that you will love it 🙂

The problem of friction of mobile AR

Mobile augmented reality has many problems that are hindering its success and one of them is the necessity to download applications to enjoy it.

One of the classical showcase augmented reality apps are AR business cards. I did one myself some years ago, even using Metaio, a framework that 4-5 years ago disappeared because it has been acquired by Apple. You had to download my app, then you could point your phone at my business card and see a textured cube popping out of it! Amazing, isn’t it?

The problem of this kind of showcase apps (apart from being useless) is friction: in my case, you had to download an app from the store, open it, frame my business card with the camera of your phone, just to see a cube with my face popping out!

The process was cumbersome, and also left an app on your phone… an app that was just useful to see a cube pop up from my business card… pretty nonsense, wasn’t it?

If only there were a way to try AR without installing an app…

WebXR to the rescue

Actually, there is a way and it is called WebXR. Thanks to this technology, it is possible to enjoy augmented reality without the need to install any kind of applications on your phone.

As a developer, you have to create a webpage using the various frameworks through which you can offer augmented reality (A-frame, the 8th Wall, ARCore, AR.js, etc…), then just provide a link to the person that you want to experience AR and bam, he/she can try your AR experience just through his/her browser without having to install anything!

WebXR removes some friction from the above process (the part where the app has to be installed)… but anyway leaves a lot of work to be done: the viewer has still to open his/her browser, write down the link, frame your business card and then finally he/she can enjoy your animation.

How to solve this problem? Well, a guy named Nicolò Carpignoli has proposed a very smart solution that employs AR.js to create a completely seamless AR experience, and experience you will have time for 😉

(It’s interesting to notice that the proposed solution can also be adapted to other frameworks, but I discovered it with AR.js, so I will show you how to develop it with AR.js)

What is AR.js?

AR.js is a Javascript framework that lets you offer augmented reality through your browser, created by Jerome Etienne and maintained by Nicolò Carpignoli.

It is amazing because it is:

• Completely opensource, with MIT license. This means that you can do with it whatever you want (even commercial applications);
• Updated frequently;
• Compatible with all the browsers and all the devices;
• Pretty fast, since it can perform at 60FPS easily;
• Very easy to be used, especially because it can work with A-frame. Creating an AR experience may be a matter of creating a webpage with 10 lines of code!

Having tried it, I can tell you that it is truly interesting. It’s not perfect, though and there are also various shortcomings that I will highlight throughout the article, this way you can evaluate if it is the right solution for you.

Marker detection

The preferred way to use AR.js is through markers. While technically it may work also without them, thanks to ARCore and ARKit, using markers guarantees that your experience is compatible with all phones.

Returning to using markers may seem a thing of the past, and in some contexts, it is not an acceptable solution, but in other cases you may need a marker to show the user that there is an augmentation in a particular place. As we will see shortly, markers can also be useful to have a good AR experience.

Let’s see what kind of markers AR.js can work with.

Barcode markers

Barcode markers are the standard ARToolkit markers and are composed by a matrix of white and black blocks. They are completely unsexy and make you feel to return to the 90s, but their detection is very reliable.

You can generate barcode markers by using this online tool. When you have to create it, you can leave all the parameters of the online tool as they are, and set only the final code (in the “Generate a single marker image with code” field). Write a number and then press Generate. The tool will create the image for you, and you can save it to the disk. Please remember with what code you have generated the marker because it will be useful later on.

Pattern marker

Since barcode markers are pretty horrible, AR.js also offers you the possibility of using pattern markers. Pattern markers are custom markers, comprised of a thick black border and a custom image inside.

Unluckily, you don’t have the freedom to make the marker as you wish, but you will have to follow these guidelines:

• The marker should use only two colors: black and light gray (#F0F0F0). Nicolò advices not to use white color;
• The marker must have a thick black border, and it is better if it also has a whitish border outside the black border;
• The shape must be square;
• The content of the marker must not be symmetrical at all;
• The marker will be interpreted by the system as a 16×16 image, so make sure to create it so that it features thick lines that can fit a similar low resolution. Big icons, letters written with big fonts, and similar, are all ok to be used as markers. Thin lines are not;
• Due to the low resolution, if you want to use multiple pattern markers, be sure that they are very different the one from the others, otherwise the system will confuse them.

You may wonder what I mean with the fact that the marker is interpreted as a 16×16 image: well, the marker can’t be used as-is, but it must be converted in a format that is readable by the detection algorithm. In this conversion, the image gets converted in a matrix of ASCII characters, in a representation just memorizes a very low resolution of the marker.

If you want an example of marker that I created… well, this is something I did to experiment with AR.js:

As you can see, the marker features very thick figures, it is not symmetric (I added the writing The Ghost Howls on the left to help with that), it is gray and black, it has a black border and a whitish border outside it. If you want some inspiration on some possible custom markers, look on Google Images for “Artoolkit marker”.

If you want some help in creating the marker, I strongly recommend you to use this online tool. Thanks to it, to create a custom marker you will have just to:

• Draw using Photoshop the square content of the marker (the gray and black ghost in the image above), and save it as a PNG. This image must be square… as dimensions, 512×512 or 1024×1024 are fine;
• Go the webpage of the online tool;
• Select Pattern Ratio: 0.5;
• Select Image Size: 512px (you can also change this if you want);
• Hit “Upload”;
• Select the image that you want to put inside the marker, the one that you created in the beginning;
• Let the system add the borders to your image;
• Download the created Image by clicking on “Download Image”;
• Download the “compiled” pattern by clicking “Download Marker”. It should be a “.patt” file.

From my experience, while barcode markers are always detected very well, custom markers sometimes don’t perform great. Sometimes the system doesn’t detect them, or confuses one marker for the other one. Be sure to follow all the above guidelines to maximize your detection rate.

How to develop an AR.js app using A-frame

Developing in AR.js, if you use Mozilla’s framework A-frame, is incredibly easy and just requires a few lines of code.

In this section, I won’t detail you everything about A-frame and AR.js and all the customization options, because otherwise this post would require days to be read, but I will just show you some heavily commented samples to show you how to use AR.js with A-Frame. At the end of the article, I will also provide you some reference articles so that to let you can learn more about these topics.

Patter markers (custom markers)

Let’s make a web AR experience that shows a red cube when it sees the above ghost marker. This is the code that you can use to do that. I have commented it a lot, so that you can understand what every line of code does:

If you remove all my comments from the above code, you obtain a very short program in which you just:

• Load the dependencies;
• Define the marker, using the <a-marker> directive, that points to the “compiled” version of the marker obtained above ;
• Define the AR content to show (in this case a cube).

And voilà, you get AR on your browser! You can test the app at this link: https://skarredghost.com/AR/red.html . You can print the marker with the ghost depicted above (here a link to the image), or you can open the image on your PC screen and use the smartphone to picture it. You should see a red box popping out.

https://gfycat.com/sociableunsteadyarchaeocete

As you can see from the video (that I shot with the webcam of my PC), the tracking is far from perfect: the virtual elements slide a bit on the marker and are not fixed in position. Furthermore, sometimes it happens that the detection doesn’t work… before I could shoot the above video, the system didn’t recognize the marker for like 5 seconds, then magically it began tracking. Even during the detection phase, sometimes when I moved the camera, the tracking got lost sometimes. Frameworks like Vuforia are much stabler than AR.js at the moment.

This may also depend on the fact that I’m trying to detect the markers that are not printed but are on the screen. To have the best detection rate, you have to print the markers, because having it on the screen makes the detection more difficult for the algorithm. I have to tell you that I had problems detecting printing markers as well.

Anyway, it’s amazing that you can obtain AR on your custom marker with just few lines of code!

Barcode markers

To use the old-school barcode markers, you use exactly the same procedure as above. There are only two differences:

• You have to add the detectionMode and matrixCodeType parameters to the a-scene;
• You have to define the marker always with the <a-marker> directive, but this time specifying the type “barcode” and writing the code associated with the barcode.

This is the program above modified so that it can show a red box on the ghost marker (image here) and a yellow box on the barcode marker with code 2 (image here).

You can test this application by accessing the page https://skarredghost.com/AR/yellow.html with your browser. Barcode markers usually get detected better than pattern ones and also tracking is slightly better, but I have noticed that the content slides also with these ones.

https://gfycat.com/defenselessgracefulaxisdeer

How to develop an AR.js app using three.js

A-frame is great to develop AR solutions that just require few lines of code. This is because A-frame is a high-level framework, that is actually built on top of three.js, a Javascript library used to create 3D applications.

You can also choose to develop AR.js experiences using directly three.js, if you have the expertise. The advantage is that the experience will be more reactive, and everything will be faster and will work better. The disadvantage is that the code will be much longer.

I’m not an expert of three.js at all, so I’ve just taken a sample by Nicolò and modified it so that it work even outside the GitHub environment it is in. You can use it as a boilerplate to start developing your experiences.

If you want to customize the code, just look for the comment beginning with “//HERE”, it will point you to the position where you can insert your custom marker. The final part of the sample is used to showcase that amazing animation that you can see on your screen when you point the marker of the ghost. You can access this sample by following this link https://skarredghost.com/AR/violet.html and framing the ghost marker.

https://gfycat.com/jointalarmedcornsnake

Personally, I love the simplicity of A-frame, so I will stick developing with it.

How to make the AR experience seamless

Ok, until now I have shown you how you can develop an AR web page using AR.js… a framework that can detect rough black and white markers. But at the beginning I promised you that this experience would have been seamless and we have still the problem of providing the link to the user. How do we solve all of this?

Well, these markers are black and white. Do you know what else is black and white? QR Codes. And QR Codes are able to contain links.

If we could combine the QR codes with the markers, the experience would be much smoother: the user should only open a QR-codes-scanning app (or the camera app in some new smartphones), that would detect the link from the QR Code, and open the related AR webpage in the web-browser. The app would decode the marker and start the AR experience immediately! Can you see how it is frictionless? The user just pictures the QR code, then some magic happens, and then he/she can enjoy augmented reality! And this exploits QR codes, that in countries like China are overly popular.

But how can we do this? There are two ways.

Put the marker inside the QR-Code

QR Codes can contain a logo at their center. If instead of a logo, we insert a marker, preferably a barcode marker, that considering the cluttered introduced by the surrounding QR code are a better choice, we can do the trick. The user will frame the QR Code and then the webpage will track the little marker embedded inside it.

The advantage of this solution is that the result is a clean QR code, with an additional code inside. The advantage is that the marker occupies only a little surface inside the code (and this is not good for the tracking stability), and you can’t add your logo to the QR code anymore.

To create the QR code, I advice you to use QRCODE MONKEY online tool. the steps that you have to perform are:

• Generate the barcode marker as we have seen above and save it to your disk;
• Use Photoshop to add a white border around it (the result should be like this one);
• Head to Qrcode Monkey;
• Select the URL tab, to tell the system you want to create a QR-code embedding a URL;
• Specify the URL of your AR webpage, in my case https://skarredghost.com/AR/yellow.html;
• Click on ADD LOGO IMAGE;
• Hit Upload Image;
• Select the image of the barcode marker that you created above and confirm;
• Check the “Remove Background Behind Logo” flag, so that there will be less clutter around the marker, improving its tracking possibilities;
• Hit Create QR Code on the right part of the screen;
• Hit Download PNG to download the image of your QR Code, so that you can distribute it.

Voilà, you obtained a QR Code that will offer an augmented reality experience to your users! Imagine scattering these QR Codes all around a museum to showcase the 3D models of the various elements exposed there!

In the video below, you can see more or less what is the experience offered by this system. On my phone, I have a QR code scanner that asks me if following the link or not, but many apps don’t have this pop-up, so the transition is immediate.

https://gfycat.com/cheerfulheartyafricanporcupine

Put the QR Code inside the marker

We can create custom markers, so why can’t we add a QR Code inside our marker design?

In this case, the procedure is this one:

• Head to Qrcode Monkey;
• Select the URL tab, to tell the system you want to create a QR-code embedding a URL;
• Specify the URL of your AR webpage, in my case https://skarredghost.com/AR/index.html;
• Select SET COLORS;
• Choose Single Color;
• As Background color, specify #F0F0F0;
• If you want, add the logo of your company, use the ADD LOGO IMAGE section;
• Hit Create QR Code on the right part of the screen;
• Hit Download PNG to download the image of your QR Code;
• Open Photoshop and create your custom marker, that is a light-gray and black image;
• Insert the just generated QR Code somewhere in your marker;
• Save the image;
• Use the online Marker Training tool as we have seen above, in the section relative to Pattern Markers, to compile the marker and download the resulting marker image and compiled patt file;
• Use the custom marker in the code as we have seen above.

The resulting marker will seem less polished, because it will be a drawing with a QR code inside, but the tracked surface will be quite big, so the tracking should perform better.

The final experience with a 3D model

The above QR code links to the page https://skarredghost.com/AR/index.html that I haven’t described you yet. You can point your phone at the marker, and see a dancing robot!

Here you are the commented source code for it. It is similar to the first sample with the red box, but it also shows you how to load a model inside A-frame:

And a video with the results:

https://gfycat.com/deaddistortedcorydorascatfish

Conclusion

We have seen how using AR.js, QR codes and a little creativity, it is possible to create an experience to show an augmented 3D model that is almost frictionless for the user. The AR.js framework is not perfect: the tracking is not stable when you move and sometimes it may be a bit unreliable, but the possibilities for the future, when these problems will be fixed, are amazing.

References

If you want to dig further in the topics discussed today, you can check out these external references:

• Nicolò Carpignoli presentation of AR.js (this is where I started);
• 10 tips to enhance your AR.js app;
• AR.js repository and documentation;
• A-frame documentation.

One more thing…

I think that these marker with embedded QR Codes have an incredible potential. If you want a solution employing them for your company, your exhibition or your museum, feel free to contact me. We of New Technology Walkers can develop it for you!

If instead you are a developer and liked this tutorial, share it with your peers on social media, and consider making a donation on Patreon to support me in offering free tutorial like this one!

(Special thanks to Enea Le Fons for having made me discover AR.js!)