How to build your passthrough AR app for Oculus Quest in Unity

I love passthrough augmented reality, because it lets us create AR experiences with the VR headsets that we already own, plus it also opens up augmented reality to new possibilities, like wide-FOV vision or diminished reality experiences. You all know that Facebook has recently released Passthrough APIs to let developers create AR experiences on Quest 2, and of course, I am experimenting with them. For instance, I’ve been doing cool stuff like this experiment in mixing AR passthrough, music, and haptics. and we of New Technology Walkers have launched our game HitMotion: Reloaded in passthrough MR on SideQuest (try it here)!

Since I love this, I would like to teach you how you can create such experiences yourself as well! So let me show you how you can develop an AR app for Oculus Quest 2 using Oculus Passthrough APIs in Unity. In this tutorial, you are going to replicate the fantastic The Unity Cube experience, this time in AR!

Oculus Passthrough APIs video tutorial

As usual, I have shot a cool video where I show you step by step how you can create an AR app for Quest 2 in Unity starting from a blank project. I think it’s a pretty cool and complete tutorial, and you can find it here below.

UPDATE (2021.12.02): The procedure in the video is still 95% correct, but the latest update to Oculus APIs has introduced some modifications. Since Youtube doesn’t let me update the video, I can just update the textual version of the article. If you watch the video, be sure to check the lines labeled with “UPDATE” (like this paragraph) in the article to see what’s changed.

If you don’t like videos, instead, keep reading for the textual version.

Oculus Passthrough APIs textual tutorial

Create and configure the project

Let’s build your first passthrough AR app together! Before going on, I warn you that while I’ll guide you step by step, I assume you know at least the basics of Unity and Oculus Quest development. If you are a complete beginner, I advise you to watch the above video, where I record myself while I perform all the operations. That said, let’s start!

Open Unity (I’m still on Unity 2019.4 LTS), and create a new blank project (URP and standard pipeline are both fine), calling it “UnityCubeMR”.

Go to Edit-> Project settings… select XR Plugin Management and click on the button to Install XR Plugin Management. This is important to enable proper VR support for your project.

When Unity has finished doing its stuff, select the “Oculus” checkbox in the Plugin Providers list of XR Plugin Management. Do it at least for Android… but if you want, you can do that also for PC.

Close the project settings and open the Build Settings going through File -> Build Settings… and switch to the Android build platform by clicking on Android and then on the Switch Platform button. Wait for Unity to do its boring stuff.

Import the official “Oculus Integration” (at least v31) from Unity Asset Store into your project. When Unity asks what to import from the package, the only things that are fundamental are the directory “VR” and the file OculusProjectConfig.asset. If you want to also have some passthrough samples to study and some helper shaders and scripts, also import the subdirectory SampleFramework/Usage/Passthrough. All the other files and directories of Oculus Integration are not fundamental for us, so import them into your project only if you need them. After you have selected the files to import, if a popup asks you to Update something always say “Yes”, and if a popup asks you to restart Unity, say “Restart”. Trust me this is de wei.

UPDATE (2021.12.02): If you want to publish your app in the Store or App Lab, make sure to have at least v34 of the SDK.

Ok, now after the editor has restarted, we are ready to tweak the SampleScene scene that Unity has created by default and transform it into a marvelous AR cube!

First thing first, delete the main camera from the scene and then add into the scene the prefab Assets/Oculus/VR/Prefabs/OVRCameraRig, that is basically the Oculus stereo camera. Verify that OVRCameraRig is in the origin for this tutorial (this is not a constraint for a generic application).

Click on OVRCameraRig and in the inspector activate the support for experimental Passthrough vision. To do that, in the OVRManager behaviour check the options “Experimental Features Enabled”, “Passthrough Capability Enabled”, and then “Enable Passthrough” (that should become available when you activate the other two). You can also activate these options in the file OculusProjectConfig.asset if you wish, but it is more handy to do that directly on the camera rig.

UPDATE (2021.12.02): Since v34 of the SDK and the runtime, the Passthrough is not experimental anymore. So you have just to check the flags “Passthrough Capability Enabled” in Quest Features -> General and “Enable Passthrough” in Insight Passthrough, all of this in the OVRManager on the OVRCameraRig.

Ok, now we have enabled the passthrough, but this is not enough to see the passthrough in your application. To do that you have to add in your scene the script that enables the passthrough level. So on the same gameobject OVRCameraRig, click on “Add Component” in the inspector and add an OVRPassthroughLayer behaviour. This is the script that will add the passthrough as a special OVROverlay at runtime.

OVRPassthroughLayer has various options to configure the passthrough.

Some relevant ones are:

• Placement: set it to “Overlay” to put the passthrough in foreground, “Underlay” to show it as the background of your application
• Opacity: useful to show the passthrough as semitransparent
• Edge Rendering: lets you underline the edges of the image, to create some trippy effects
• Color Map: lets you define a mapping between the black/white passthrough and an RGB color gradient. It is useful to color the passthrough with the colors that you want, so it doesn’t appear only black and white.

Tweaking these parameters you can create some nice effects. In the below tweet, for instance, I modified them to show my world as always black, and have an echo location visualization of my surroundings depending on the intensity of the music I was playing:

https://twitter.com/SkarredGhost/status/1440426629717626881

For this demo, select:

• Placement: Underlay
• Edge Rendering: Activated
• Edge Color: RGBA(0, 209, 255, 255)

Basically, I set the passthrough vision as background of my scene (so the virtual elements appear on the foreground), and I specified that I want azure edges highlights.

At this point, theoretically we have set everything to have an AR experience on the Quest 2, but actually if you tried to build the program in this moment, nothing would be working. It’s because Oculus needs you to specify these settings to make passthrough to work:

• Go to Windows -> Rendering -> Lighting Settings and remove the skybox: click on Skybox Material and hit CANC on your keyboard. Skybox material is rendered below all foreground elements as well, and sometimes it appears in front of passthrough, occluding it. So either you remove the Skybox or you play with its shader (and render queue) to put it behind the passthrough images… otherwise you risk not seeing your passthrough underlay

UPDATE (2021.12.02): A valid alternative to removing the skybox is selecting the Main Camera of your scene (the gameobject CenterEyeAnchor inside the OVRCameraRig) and set in the inspector the “Clear Flags” (“Background Type”, if you are using URP) as “Solid Color” and then specify as Background Color (0, 0, 0, 0), that is transparent black.

• Go to Oculus -> Tools and select OVR Performance Lint Tool. Enlarge a bit the dialog that pops up. Expand the “Quest Issues” label and click the “Fix” buttons of the following voices, in this exact order: “Optimize Scripting Backend”, “Set Android Target SDK Level”, and “Set Target Architecture to ARM64” (this as the last one). Doing this we enable IL2CPP and ARM64 support, that are also required to publish on the store nowadays

• Close the Lint tool, and head to the project settings (Edit -> Project Settings…). In the Player tab, at the Other Settings section, specify Color Space as Linear. At this point, Unity may reimport all the resources
• Close the project settings and head to Oculus -> Tools -> OpenXR -> Switch to OVRPlugin with OpenXR backend. Click on it: at this point if you are asked a confirmation say Ok, then confirm to restart the Editor (click “Restart” on the popup, if any) and when asked if your current scene has to be saved, say “Yes”. Wait for Unity to restart. Activation of OpenXR is fundamental because Facebook is dropping support for the native OVRPlugin backend in its runtime, and all the new features are implemented only with the OpenXR backend. So if you don’t enable OpenXR, you can’t have passthrough. Kudos to Oculus that is switching to this open standard that will let us build once and deploy everywhere.

At this point, the project is properly configured for passthrough. This is the only configuration that I found that guarantees passthrough vision in your application. I tried activating only OpenXR, or only ARM64, and I could see the passthrough only as a full black background. Only activating all of them everything worked, showing me my surroundings. Maybe these steps are not all necessary, but this is the only configuration that I can guarantee you that always work. And I have to thank a lot Dilmer Valecillos for his amazing tutorial that helped me in understanding the need for some of these flags: head to his great Youtube channel if you need to learn something about XR development in Unity!

Now that the project is all set, we can finally craft our scene!

Design the Unity scene

First of all, let’s add controllers in your hands: take the prefab OVRControllerPrefab that you find in Assets/Oculus/VR/Prefabs and put one instance as child of OVRCameraRig/TrackingSpace/LeftHandAnchor/LeftControllerAnchor in the Hierarchy. Select it and in the inspector, set Controller as L Touch in the dropdown. Add another instance of that prefab as child of OVRCameraRig/TrackingSpace/RightHandAnchor/RightControllerAnchor, setting Controller as R Touch.

Now, let’s add the cube! Click on the “+” button in the hierarchy panel, select 3D Object -> Cube. Set its position as (0, 0, 1.5) so that it stays in front of you.

Save the scene. Everything is done!

We are ready to build. Select File -> Build Settings… Click “Add Open Scenes” in the dialog, and then select “Build”. Don’t do build and run for now… because the app wouldn’t run properly (yeah, this passthrough is a bit a nuisance). Specify the name of the APK and give it some minutes to build it (IL2CPP builds are pretty slow)

Run the experience

Before running the experience, be sure that your Quest runs v31 or v32 runtime, or passthrough won’t work. If your firmware is too old, update your Quest until you reach the latest version. And be sure that you have a Quest 2, at the time of writing, Quest 1 has not passthrough enabled.

UPDATE (2021.12.02): I strongly suggest having runtime v34, so that Passthrough is not considered experimental anymore. All the below commands are now obsolete and you don’t need to perform the below operations if you have v34 or greater version. I leave them here below in case someone has still an old version of the runtime.

I strongly suggest you to install SideQuest to perform the next operations, but you can also use plain ADB commands, if you wish.

Connect the Quest to your PC via USB cable. We need the connection not only to install the APK you have just built, but also to enable Experimental features. Passthrough is currently considered an experimental feature and so to use it, you have to enable Experimental Mode on your device, and this must be done AFTER EVERY REBOOT OF THE HEADSET.

Open SideQuest, and click on the Wrench button in the upper right corner of the window (its tooltip is Device Settings & Tools). Look for Experimental Mode section and click the button “ON”. SideQuest should confirm it with a green stripe saying that the mode has been activated. If the stripe is yellow and shows an error, keep hitting the ON button until it works (yeah, the good old bruteforce way always works)

If you don’t want to use SideQuest, the ADB command to activate this mode is “adb shell setprop debug.oculus.experimentalEnabled 1”.

UPDATE (2021.12.02): All the previous commands are not necessary from v34 on. Just install and run the app, and you are set to go!

After this, we are ready to install the APK! If you are using SideQuest, click on the little 9-squares-grid symbol in the upper right corner, and then drag the APK you have just built from the File Manager to the SideQuest window, and wait for it to confirm the installation. If you want to use plain ADB, use the command “adb install -r <PATH_TO_APK>”.

You can now run the app! Put ON your Quest and find your application in the Unknown Sources tab, with the name “UnityCubeMR”. Selecting it, you should see a fantastic cube in your room, with the visuals of your room in black and white with light blue edges!

https://gfycat.com/idioticgrosseyra

Bonus Info

Here you are some special bonus info for you on Passthrough APIs:

• If you are using XR Interaction Toolkit, so an XRRig gameobject instead of an OVRCameraRig, you can still use Passthrough. First of all, be sure that the main camera in the XRRig has the tag MainCamera. Then you add an OVRCameraRig prefab to your scene as before, setting the same exact settings as before. The only difference is that in the behaviour OVRCameraRig of that gameobject, you should set the “DisableEyeAnchorCameras” flag. Also disable the “Post Processing” flag on the main camera under the XR Rig (Currently, it makes the passthrough become black… I have spent two days to find how to solve this issue!). This way, you are good to go. Oculus will use its prefab to handle the passthrough, and XRRig will be used as the main camera for your visuals;
• Inside Unity you have NO WAY to see or to access the camera frames. Literally no way, because the passthrough images are added by the operating system when the app is running. So even if you add a secondary camera that renders your scene to a RenderTexture, it can’t picture the passthrough images. Read more about this in my Passthrough review;
• Many tutorials say that to use a passthrough application, you should have the Guardian enabled. This is not true anymore, and you can disable the Guardian and go all around your city with passthrough vision activated, as I dimonstrate in this other post;

https://gfycat.com/bleakfatherlyleveret-virtual-reality-mixed-reality

• To record footage of you Passthrough AR application you can use SideQuest, as I describe in this post;
• If you want to learn more about Passthrough APIs, you can read the official documentation here: https://developer.oculus.com/experimental/passthrough-api/
• You can also experiment more by analyzing and modifying the sample scenes that are in /Assets/SampleFramework/Usage/Passthrough/Scenes

UPDATE (2021.12.02): You can start with passthrough disabled (the check “Enable Passthrough” on OVRManager left unchecked) and activate it later by calling “OVRManager.instance.isInsightPassthroughEnabled = true” in your code. But in my tests, if you keep changing the value of “OVRManager.instance.isInsightPassthroughEnabled”, sooner or later, everything crashes. So my suggestion if you want to enable/disable passthrough in your app many times is to leave the option enabled and then use “FindObjectOfType<OVRPassthroughLayer>.hidden = true/false” to activate and deactivate it.

---

And that’s it! Now you have all the info you need to start building your augmented reality apps for Oculus Quest 2. Experiment with it, and let me know what you create with it! And share this article with your peers so that also all of them can experiment with Passthrough APIs, too!