Vivi Wu Posted May 25, 2023 Share Posted May 25, 2023 Introduction The “VIVE Focus 3”, “VIVE XR Elite” and “VIVE FOCUS vision” provide camera passthrough features, especially “VIVE XR Elite” and “VIVE FOCUS Vision” have a great color passthrough. More and more content developers look forward to use the camera passthrough to create an immersive MR content. Now, Vive Business Streaming brings this vivid passthrough to PC content with MR support. In this document, we would like to mention that VBS can now enable passthrough underlay on VIVE HMDs during streaming with just a click. OpenVR content developers can easily adapt their applications for an immersive mixed reality experience using VBS's MR mode. By submitting the alpha layer to the SteamVR Runtime—a feature many existing applications, such as Blender, already support—developers can enable MR with minimal changes. For OpenXR content developers, we offer additional support through the passthrough extension. Please refer to the tutorials below for more details. VBS will take care of all the rest work to stream the alpha layer and blend with camera passthrough at headset side to create MR experience. PC users can now enjoy Mixed Reality contents with the high-quality color passthrough on VIVE XR Elite and VIVE FOCUS vision, also worked on VIVE Focus3. OpenXR We support XR_HTC_passthrough (The OpenXR™ Specification (khronos.org)) extension in VBS, please refer to below URLs for more information. To use this extension, you won’t need to enable MR option manually. Requirements: VIVE Streaming Hub v2.0.0a / VIVE Business Streaming v1.15.11 or newer Supported GPU: NVIDIA and AMD Tutorials: Unity- https://developer.vive.com/resources/openxr/unity/tutorials/passthrough/ Unreal Engine- https://developer.vive.com/resources/openxr/unreal/unreal-tutorials/passthrough/ OpenVR Please refer to below guidelines to enable MR mode in OpenVR contents. Requirements: VIVE Streaming Hub v1.1.9 (developer only) / VIVE Business Streaming v1.11.3 or newer Supported GPU: NVIDIA and AMD To enable MR with passthrough VIVE Business Streaming: Open VBS Console Settings Switch to “Graphics” page and enable “MR with passthrough” Launch MR content and enjoy You can enable/disable MR mode in anytime VIVE Streaming Hub To enable developer mode, download devmode.jsonfile and copy to C:\Program Files\VIVE Hub\ folder. Start VIVE Streaming Hub, then the MR with passthrough option should be there in VIVE Streaming settings. Enable MR with passthrough option. Develop your MR content for VBS VBS extracts the alpha channel of textures of Content and use the information to blend VR texture and passthrough. If the content developers would like to mix real environment and content scene, they have to assign alpha value into the scene. Please read below steps for more details. The content textures format should include ALPHA layer The alpha value should be 0 to 1. To show passthrough on the HMD, please set the alpha values closed to 0 into target area in the scene. To show the content texture on the HMD, please set the alpha values closed to 1 into target area in the scene. 1. Content textures 2. Extracted Alpha layer texture 3. MR result from HMD Camera setup in Unity How to set transparent camera background in Unity: Select the Main Camera in the Hierarchy panel In the Inspector panel, find the Camera component Change the Clear Flags to Solid Color Change Background color to RGBA values (0, 0, 0, 0) You can use Unity's Color.clear to set this in your code Troubleshooting If there is no passthrough textures while MR mode is enabled, how to recover? Please reopen the function again to check if the MR mode normal Please verify the alpha layer of the textures is already provided with our steps and submit to SteamVR. The performance looks worse when MR mode is enabled. Please check the Console settings -> Graphics settings -> Streaming graphics preferences, we suggest not to use Ultra preference when MR mode is enabled due to additional encoding involved. How to record the MR content result in headset? Please make sure “Allow passthrough recording and casting” is enable. You can find the setting from HMD’s Lobby, Settings -> Advanced -> Camera settings. When you change this item, please reboot your device. Link to comment Share on other sites More sharing options...
Jerome Posted June 13, 2023 Share Posted June 13, 2023 Hello, I tested with a simple UE 4.26 project with a vive Focus 3. Don't know if it's a problem with Unreal gine and this version but it seem Blending is done on the color intensity (if an object is white it will be "opaque", if it's 50% grey it will 50% transparent. I don't know if there the right way to pass alpha with UE ? And VR part / camera view seem not be in the same coordinate, there is a lot of parralax between VR and reallity, object doesn't fit were they should, very strange effect, I will try with UE 5.2 Link to comment Share on other sites More sharing options...
CREWvzw Posted June 28, 2023 Share Posted June 28, 2023 On 6/13/2023 at 1:00 PM, Jerome said: Hello, I tested with a simple UE 4.26 project with a vive Focus 3. Don't know if it's a problem with Unreal gine and this version but it seem Blending is done on the color intensity (if an object is white it will be "opaque", if it's 50% grey it will 50% transparent. I don't know if there the right way to pass alpha with UE ? And VR part / camera view seem not be in the same coordinate, there is a lot of parralax between VR and reallity, object doesn't fit were they should, very strange effect, I will try with UE 5.2 Hey there, Could you elaborate on how you exactly achieved this in Unreal Engine? The pipeline is a bit unclear to me. Which plug-in do we use, SteamVR? And how do we submit the alpha layer to SteamVR? 1 Link to comment Share on other sites More sharing options...
jiunlin Posted July 3, 2023 Share Posted July 3, 2023 You can try in Project Settings > Rendering > Postprocessing, set Enable alpha channel support in post processing to Allow through tonemapper. 1 Link to comment Share on other sites More sharing options...
lynXR Posted July 10, 2023 Share Posted July 10, 2023 Hello, Can I know which plug-in do we use if we want color passthrough for the PCVR project with the device XR Elite in Unity3D? Do you have VIVE OpenXR API for that or we need use steamVR SDK? 2 Link to comment Share on other sites More sharing options...
Ilija Posted July 19, 2023 Share Posted July 19, 2023 On 7/10/2023 at 3:28 PM, lynXR said: Hello, Can I know which plug-in do we use if we want color passthrough for the PCVR project with the device XR Elite in Unity3D? Do you have VIVE OpenXR API for that or we need use steamVR SDK? Stuck on the same page here, hope the devs answer soon Link to comment Share on other sites More sharing options...
Vivi Wu Posted July 20, 2023 Author Share Posted July 20, 2023 Hi @Ilija @lynXR, The change on the content is to modify the textures of ALPHA layer. So you could use OpenVR or OpenXR SDK for your PC content on Elite or Focus3. Link to comment Share on other sites More sharing options...
Ilija Posted July 20, 2023 Share Posted July 20, 2023 3 hours ago, Vivi Wu said: Hi @Ilija @lynXR, The change on the content is to modify the textures of ALPHA layer. So you could use OpenVR or OpenXR SDK for your PC content on Elite or Focus3. Ok, now I have made it and feel rather stupid because it didn't work just until i updated the streaming app on the headset.@Vivi Wu , thank you very much for your help and tutorial, you are a life saver! Link to comment Share on other sites More sharing options...
Ilija Posted July 24, 2023 Share Posted July 24, 2023 @Vivi Wu Hello again, I have another question. You see, I managed to make it work with Vive Business Streaming on Focus 3, but the workaround with devmode json doesn't seem to work with Vive Streaming Hub. Updating VSH to beta also doesn't seem to help much -- the passthrough option appears but the app itself shuts down after a few seconds of working. Do you have an advice for that? Help will be much appreciated, thank you in advance! Link to comment Share on other sites More sharing options...
Ray Su Posted July 25, 2023 Share Posted July 25, 2023 hello, is it possible to get passthrough result in webxr, or how can i get the origin result of passthrough i'd want to implement mr in web by myself, but i don't know where to get the passthrough result Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now