[PC VR/VBS] VIVE Streaming MR with passthrough

[Vivi Wu]

Introduction

The “VIVE Focus 3” and “VIVE XR Elite” provide camera passthrough features, especially “VIVE XR Elite” has a great color passthrough. More and more content developers look forward to use the camera passthrough to create an immersive MR content. Now, Vive Business Streaming brings this vivid passthrough to PC content with MR support. In this document, we would like to mention that VBS can now enable passthrough underlay on VIVE HMDs during streaming with just a click.

Content developers can effortlessly transform their content into immersive mixed reality encounters to work with VBS MR mode. By submitting alpha layer to SteamVR Runtime which many contents already support, e.g. Blender. VBS will take care of all the rest work to stream the alpha layer and blend with camera passthrough at headset side to create MR experience. PC users can now enjoy Mixed Reality contents with the high-quality color passthrough on VIVE XR Elite, also worked on VIVE Focus3.

Requirements

VIVE Streaming Hub v1.1.9 (developer only) / VIVE Business Streaming v1.11.3 or newer
Supported GPU: NVIDIA and AMD
Supported SDK: OpenVR and OpenXR

To enable MR with passthrough

• VIVE Business Streaming:
  1. Open VBS Console Settings
     
  2. Switch to “Graphics” page and enable “MR with passthrough”
     
  3. Launch MR content and enjoy
  4. You can enable/disable MR mode in anytime
• VIVE Streaming Hub
  1. To enable developer mode, download  devmode.jsonfile and copy to C:\Program Files\VIVE Hub\ folder.
  2. Start VIVE Streaming Hub, then the MR with passthrough option should be there in VIVE Streaming settings.
  3. Enable MR with passthrough option.

Develop your MR content for VBS

VBS extracts the alpha channel of textures of Content and use the information to blend VR texture and passthrough. If the content developers would like to mix real environment and content scene, they have to assign alpha value into the scene. Please read below steps for more details.

1. The content textures format should include ALPHA layer
2. The alpha value should be 0 to 1.
3. To show passthrough on the HMD, please set the alpha values closed to 0 into target area in the scene.
4. To show the content texture on the HMD, please set the alpha values closed to 1 into target area in the scene.

 

1.     Content textures
2.     Extracted Alpha layer texture
3.     MR result from HMD

 

Camera setup in Unity

How to set transparent camera background in Unity:

1. Select the Main Camera in the Hierarchy panel
2. In the Inspector panel, find the Camera component
3. Change the Clear Flags to Solid Color
4. Change Background color to RGBA values (0, 0, 0, 0)
   1. You can use Unity's Color.clear to set this in your code

Troubleshooting

• If there is no passthrough textures while MR mode is enabled, how to recover?
  • Please reopen the function again to check if the MR mode normal
  • Please verify the alpha layer of the textures is already provided with our steps and submit to SteamVR.
• The performance looks worse when MR mode is enabled.
  • Please check the Console settings -> Graphics settings -> Streaming graphics preferences, we suggest not to use Ultra preference when MR mode is enabled due to additional encoding involved.
• How to record the MR content result in headset?
  • Please make sure “Allow passthrough recording and casting” is enable. You can find the setting from HMD’s Lobby, Settings -> Advanced -> Camera settings. When you change this item, please reboot your device.

[Jerome]

Hello,

I tested with a simple UE 4.26 project with a vive Focus 3.

Don't know if it's a problem with Unreal gine and this version but it seem Blending is done on the color intensity (if an object is white it will be "opaque", if it's 50% grey it will 50% transparent. I don't know if there the right way to pass alpha with UE ?
 

And VR part / camera view seem not be in the same coordinate, there is a lot of parralax between VR and reallity, object doesn't fit were they should, very strange effect, I will try with UE 5.2

 

[CREWvzw]

On 6/13/2023 at 1:00 PM, Jerome said:

Hello,

I tested with a simple UE 4.26 project with a vive Focus 3.

Don't know if it's a problem with Unreal gine and this version but it seem Blending is done on the color intensity (if an object is white it will be "opaque", if it's 50% grey it will 50% transparent. I don't know if there the right way to pass alpha with UE ?
 

And VR part / camera view seem not be in the same coordinate, there is a lot of parralax between VR and reallity, object doesn't fit were they should, very strange effect, I will try with UE 5.2

 

Hey there,

Could you elaborate on how you exactly achieved this in Unreal Engine? The pipeline is a bit unclear to me. Which plug-in do we use, SteamVR? And how do we submit the alpha layer to SteamVR?

[jiunlin]

You can try in Project Settings > Rendering > Postprocessing, set Enable alpha channel support in post processing to Allow through tonemapper.

[lynXR]

Hello，

Can I know which plug-in do we use if we want color passthrough for the PCVR project with the device XR Elite in Unity3D? Do you have VIVE OpenXR API for that or we need use steamVR SDK?

[Ilija]

On 7/10/2023 at 3:28 PM, lynXR said:

Hello，

Can I know which plug-in do we use if we want color passthrough for the PCVR project with the device XR Elite in Unity3D? Do you have VIVE OpenXR API for that or we need use steamVR SDK?

Stuck on the same page here, hope the devs answer soon

[Vivi Wu]

Hi @Ilija @lynXR,

The change on the content is to modify the textures of ALPHA layer. So you could use OpenVR or OpenXR SDK for your PC content on Elite or Focus3. 

[Ilija]

3 hours ago, Vivi Wu said:

Hi @Ilija @lynXR,

The change on the content is to modify the textures of ALPHA layer. So you could use OpenVR or OpenXR SDK for your PC content on Elite or Focus3. 

Ok, now I have made it and feel rather stupid because it didn't work just until i updated the streaming app on the headset.
@Vivi Wu , thank you very much for your help and tutorial, you are a life saver!

[Ilija]

@Vivi Wu
Hello again, I have another question. You see, I managed to make it work with Vive Business Streaming on Focus 3, but the workaround with devmode json doesn't seem to work with Vive Streaming Hub. Updating VSH to beta also doesn't seem to help much -- the passthrough option appears but the app itself shuts down after a few seconds of working. Do you have an advice for that? Help will be much appreciated, thank you in advance!

[Ray Su]

hello, is it possible to get passthrough result in webxr, or how can i get the origin result of passthrough

i'd want to implement mr in web by myself, but i don't know where to get the passthrough result