Using AR as a spectator view for your VR app

This is one of my favorite use cases for Normcore. Before diving into this guide, I recommend reading this blog post about how this concept works at a high level.

Before we can start, we need an example VR project that we’ll use with our AR spectator app. I’m going to start with the “Realtime + VR Player” scene from the Normcore Examples folder. Import Normcore, open up the VR Player example scene, and add your App Key to the Realtime component in the scene.

Hit Play in Unity and make sure the VR Player prefab spawns in our colored cube world.

Hello, world!

Awesome! Now that we’ve got that working, let’s get started on the mobile AR version. Use Package Manager to import AR Foundation and either the ARKit XR Plugin (iOS) or the ARCore XR Plugin (Android).

Adding AR Foundation and ARKit via the Package Manager.

Next, we’re going to import the AR Foundation Example unitypackage from Normal which includes a few utilities and the scene you’ll end up with at the end of this guide. Depending on your AR Foundation version, you’ll want to download the appropriate package.

Importing the 'AR Foundation Example' Unity package.

Perfect. Let’s save a copy of this scene to use for our AR Spectator app. Here I’m going to call it “VR Player (AR Spectator)”.

In our new scene copy, let’s turn off the Floor. In the spectator app, we’ll want to see the colored cubes and VR player, but we’ve already got a floor in real life, so there’s no reason to render a virtual one on top.

Disable the Floor object, and then save the scene.

Next, we’ll want to swap the Main Camera for one that supports AR. Drag the “AR Foundation Camera Rig” prefab from the Examples > VR Player (Spectator) > Prefabs folder into the scene. This prefab contains the scripts needed to position the AR Camera, display detected planes, and finally place the origin of the world on an AR plane. It also includes a Camera object, so we can go ahead and delete the Main Camera game object from the scene.

Drag in the 'AR Foundation Camera Rig' prefab, and delete your existing Main Camera.

The only thing left to do is connect RealtimeAvatarManager to our camera rig so the avatar uses the phone’s camera position in VR. We’ll do that by dragging the AR Camera into the Head slot on RealtimeAvatarManager.

Drag the 'AR Camera' under the AR Foundation Camera Rig into the Head transform on Realtime Avatar Manager.

That’s it! Let’s give it a shot!

Export a build for your device. When it opens up, move your phone around to detect planes, then tap on one to place the world. You should see the colored cubes appear.

Now head back to Unity and open up the original “Realtime + VR Player” scene and hit Play. Your VR player will appear in the scene view, and also on your mobile device :)

It works!

Boooooom!! Easy.

Want to take this a step further? Try creating a custom avatar for your AR Spectator so they look different than the VR players in VR. Check out our other guides for synchronizing custom data too!

FAQ

  • Xcode says linker command failed with exit code 1 (use -v to see invocation)
    • This usually means that ARKit.framework and/or libz.tbd are not linked. Make sure to add them to the “Link Binary With Libraries” section under “Build Phases” in Xcode
  • On iOS the app opens and closes immediately
    • On iOS, make sure to set your Camera Usage Description and Microphone Usage Description under Player Settings.