Android XR will allow camera access like on the phone

Passthrough camera access is a hot topic in the XR community right now and while the positions of Meta, Apple, and Pico are known, everyone is wondering what Google will do with Android XR. After I’ve spoken directly with the company, I can tell you that they will offer a solution similar to the one on the phone. Keep reading to discover more!

The camera access dilemma

Let me take a step back for all of you who are wondering what I’m talking about. We all know that all the most recent standalone VR headsets are actually MR headsets and they show the user an RGB passthrough vision of the world captured by front cameras. This allows us to have all those lovely mixed reality applications like Cubism, Starship Home, Pencil, and so on.

The operating system uses the frames captured by the cameras to show us the passthrough. We developers would love to have access to these camera frames, too, so that we can analyze the world around the user with AI and computer vision algorithms and enhance his/her reality in a powerful way. As I’ve written in a previous post of mine, I’m a strong believer that we need camera access to unlock true mixed reality, because only thanks to that our applications can be fully context-aware of the reality around us. For instance, using a hacky camera access trick on Quest, I’ve been able to prototype an AI+MR application that can help people do some interior design of their houses. This wouldn’t have been possible without it.

This sounds great, but there is a huge counter-argument: the privacy issues. If a malicious developer is granted camera access, he can for instance silently grab all the images of the environment around the user, run them through some AI detection, and scrape some precious personal information, like data about National ID documents or bank cards that the user has on the table while he uses the application. Not to mention the possibility of grabbing pictures of faces or bodies for other wrong uses.

It’s a very delicate matter: on one side it is important to guarantee the privacy of the users, but on the other side, it is important to unleash the potentialities of mixed reality.

The behavior of the XR companies

In the very early stages, there was full camera access offered without any issues. If you have followed me since the beginning, you may remember the experiments I did with camera textures on the Vive Focus with my team at NTW (it was 2019 circa): diminished reality, Aruco marker tracking, sound reactivity, etc…

Then, as soon as mixed reality started to become a thing, companies started to play safe, and all of them blocked access to camera frames for fear of privacy concerns. Meta, Pico, HTC, then Apple… all of them prevented the developers from accessing the camera frames.

This became the standard until the community of XR developers started realizing the need for this feature and started pressuring the XR manufacturers to provide camera access. Some people exposed themselves a lot in this sense, like for instance Cix Liv, Michael Gschwandtner, and me. We started asking all together to have access to camera frames in a way that was clear and transparent to the user, so that if the user agreed to that, we could run for instance object recognition or computer vision algorithms on what the user has in front of him/her. We questioned why this fear about XR devices when on the phones, which are the devices that today every one of us has in our pockets, camera access is possible with just a permission request.

This movement led to XR companies opening up to this feature, with Meta promising a “Passthrough API” to be available at the beginning of this year. But what about Google and Android XR?

Android XR to treat the headset like a phone

Android is the operating system of the majority of the phones worldwide. Currently, if you are developing an application for Android, you can request access to a camera stream by asking permission from the user. If the user grants the permission, you just have to specify the ID of the camera you want to access (e.g. usually 0 is the back camera) and then you can grab the frames and do with them what you want.

Google wants to make Android XR as compatible with Android apps as possible and in fact, it is providing a similar behavior on its new operating system. There have been many rumors about it, but no official confirmation, so I had a long email exchange with a Google spokesperson to confirm this. Let me copy-paste her answer to my request about camera access on Android XR to clarify this once and for all:

• Similar to any Android app, a developer can use existing camera frames with user permission for XR
• Our developer article goes into depth on other permissions that the apps can request: https://developer.android.com/develop/xr/get-started#understand-permissions
• A developer can request camera_id=0 for the main world-facing camera stream. This corresponds to the “rear camera” in standard Android parlance. Similarly, a developer can request camera_id=1 for the main selfie-camera stream. This corresponds to the “front camera.” Both streams are accessed through the standard Android Camera APIs, i.e. Camera2 and CameraX
• To get access to the world-facing camera feed (similar to the back camera on a smartphone), the app will have to request camera permissions (analogous to the permission requested for camera access on phones)
• When apps request the selfie-camera, they receive an image stream containing the user’s avatar. This avatar video stream is generated by Avatar provider apps/services on the headset. These apps generate the stream based on user tracking data from OpenXR APIs, e.g. head, hand, eye, face. This tracking data is obtained from the physical inward-facing cameras that monitor the user’s movements and facial expressions.

Android developers can so use the standard classes they use today on the phones to manage camera streams (like CameraX) also on Android XR-based headsets. Since these classes allow the devs to grab frames and videos, save media, or run ML analysis on them, this should be allowed on headsets and glasses, too. This is great news!

But while the front camera is accessible, the rear camera is just a reconstructed avatar, similar to what Apple is doing with the Vision Pro. I think this is a smart decision for a few reasons, with the most important of them being that this way Android XR again behaves like Android on phones: if an application accesses the “rear camera”, it sees the world the user has in front of his/her, while if it accesses the “selfie camera”, it sees the face of the user. And the best guess the OS has of the face of the user is the reconstructed avatar face.

Google is trying to make sure that all Android apps work out of the box on Android XR and the above choices make sure also the apps that access cameras can work in the same way. This is very smart. And I’m also personally a big fan of keeping coherence with the permission request between phones and headsets/glasses.

You may have still a couple of questions at this point. The first is what about having access to all the raw camera streams. For that, I have not a positive answer for you, sorry:

• At the moment, we are not providing a path for applications to access the non-standard (e.g. forward facing camera, reconstructed inward camera) sensor data.

Another good question is about Unity development. Android Camera2 and CameraX are Android native classes, so what can we Unity developers do? Well, if Android XR offers a similar path to Android, then we Unity developers can likely use the WebcamTexture class to grab the frames of the camera. In case for some weird reason, this may not work, someone can still use some JNI black magic to create a Unity-to-native wrapper library of the CameraX function to give the community the possibility of having camera access.

A little caveat about Android XR

Android XR is still in preview and no headset running it has been officially released. This means that all of the above may change before the final official release. I don’t think it is going to happen, but it is important that you consider that there is a possibility.

The opening up of camera access

With Google and Meta opening up to camera access, probably all the other companies will follow suit. 2025 so seems suited to be the year when new possibilities in mixed reality will be unlocked. I can’t wait to see what the developers’ community will create!

(Header image based on a Samsung image)