Exploiting MR passthrough in VBS-remoted PC apps

[Paolo Leoncini]

In my Focus 3 I've just experienced an apparent issue related to video passthrough mapping over semi-transparent surfaces in VBS version 1.12.8, which 1.13.2 has "solved" (the MR passthrough UI flags in the Graphics tab now works).

This was the opportunity to discover the MR video passthrough joys, which I kindly ask to explain in more detail - or point to explaining docs - in order to use it with awareness, especially along with the new XR Elite color passthrough.

To my practicality in these days, MR passthrough - when enabled in VBS - got mapped in a sort of chromakey mode over transparent and semi-transparent parts of the 3D scene. Yet, I never got "pure" passthough, in what it was always mediated by the material props (diffuse/ambient/emissive values), and this is a mistery to me because as far as I understand VBS barely streams the PC app framebuffer, thus the link to 3D stuffs should have been lost in the rasterization.

What is the best config for a pure MR passthrough mapping on wanted 3D scene surfaces?

Thank a lot in advance for every explanation that would shed light on this behavior.

Paolo

[Paolo Leoncini]

I found this: 

The problem is that the MR passthrough gets mapped wherever alpha <> 0 (i.e. where all the semi-transparent surfaes gets rasterized and put alpha non zero into the framebuffer), not only on what we could agree to define "background".

[Alex_HTC]

@Paolo Leoncini

Howdy Paolo!

I understand you've encountered an issue with video passthrough mapping on semi-transparent surfaces in VBS, and you're looking for a way to achieve a pure MR passthrough mapping on your 3D scene surfaces. Let's tackle your concerns:

Color and Alpha Settings:
1. It appears that color is utilized even when the alpha flag is set to 0. While I'm not certain about the 'best' settings, many find either pure white or gray to be preferable.

Unreal-Specific Alpha Issues:
2. There might be some unreal-specific alpha issues, which I believe have been addressed in the latest version of vbs. 

 

Best Practices:
- **General**: Tutorial for MR Contents
- **Unreal**:
   - [Unreal Passthrough Underlay] for a basic understanding.
   - [Unreal Passthrough Quality Performance] for tuning performance, scale, and position.
- **Unity**:
   - **Wave SDK**: [Unity XR Passthrough]
   - **OpenXR**: [Unity Passthrough with OpenXR]

Contextual Usage of MR:
Ensure the tracking mode is not set to 'instant mode'; room-scale tracking is preferred. Follow the MR room setup in the settings, then explore the Scene Perception SDK as specified [here], and the 'scene perception' demo for a hands-on understanding.

Occlusion Based on Mesh Scans:
Using scene perception, obtain a visual mesh `WVR_SceneMeshType.WVR_SceneMeshType_VisualMesh` as specified [here], and employ layers or other tests to cull virtual objects for occlusion.

MR Passthrough Mapping in Chroma-Key Style:
We have some helpers for this in unity. An example project is available  https://github.com/ViveSoftware/Jelbee-MR-Demo. It comprises a component to select an area and lower-level APIs for more customized use-cases. Typically, these are used in reference to real-world objects using anchors as specified [here], or for 'HUD' style setups. You can blend the two by culling/occluding objects in the real world based on the position of the visual mesh provided by the above SDKs.

I hope these resources and explanations address your concerns and help you move forward with your project!

[Paolo Leoncini]

Thank you a lot Alex, now the picture is much clearer.

For starting, in my OpenGL-based VR runtime I just added the fourth component to glClearColor(r, g, b, 0) and... le MR sont fait!

On the Unreal side I'll need to go through an investigation since in an application using deferred rendering (and RT, I don't know if this matters), frames result in being fully filled by the passthrough video - maybe UE puts an invisible transparent envelope all around the scene that becomes an alpha = 0 for the MR passthrough. For now, just disable the MR flag in VBS.

Paolo

[Alex_HTC]

@jiunlinCan you put the RT issue on your ray-dar. or do you have a response already. That the ray tracing seems to clobber the background a bit in some circumstances.

@Paolo Leoncini 1) good to hear that you found the workaround useful! 2) Related to RT - Interesting! I am unaware of any deferred rendering issues, but with my mental model of things the type of ghosting shown in the second video would definitely happen with ray tracing for several reasons, most importantly would be opaque pixels being put in front of the background image.

I'm not the authority here, but to some extent this is what i would expect ray tracing to do - put non-transparent stuff on top of the background. Of course RT is a big field, but i would expect both lighting passes on the pc and content rendering using ray tracing to have similar results. I'm sure there's some way to maybe get them to play well together with some alpha in the lighting colors -- just a wild guess from me to try to better understand in the meantime.