Chroma Keying with Vive XR Elite

[msoson]

I'm looking to build chroma keying functionality into a research application I’m building.  This would need to be through the Vive or OpenXR SDK in Unity or Unreal game engines.  I understand that Vive does not allow access to the camera image textures to developers for privacy reasons, so wanted to know if there was any planned functionality in those SDK’s for the Vive XR Elite.  The end result would be someone in headset, looking at someone else against a green background, but the background is keyed out and replaced with game space.  Thus the need for some kind of chroma key functionality in headset (not another camera, as with LIV, from my understanding) for my mixed reality experience.

[Konshu]

Could something like this work: https://youtu.be/6_wFfWSNmx0

[msoson]

Hi @Konshu.  There are many ways to chroma key a normal video or image texture in Unreal, even just using materials and not an external plugin, so that's not really the problem I'm inquiring about (though I appreciate the thought).  I'm specifically asking about access to the Vive Elite passthrough texture layer, which Vive has been previously explicit about devs not having access to (for privacy reasons), and then applying chroma keying, or really any material post processing to.

[llupiy.com]

Same here. We use the Varjo XR3 for chroma key compositing and I am looking for a more lightweight solution.

[loneill]

@msoson Did you ever end up finding a solution to this problem? I am in a similar situation and after posting a question on here, never heard back

[msoson]

@loneill @llupiy.com Closest I've come is experimentation with Zed Mini using Touchdesigner and Unity, which does work with the quest 2, though obviously not ideal.  The Lynx looks promising. https://www.lynx-r.com/

 

This was a narrative experience we made with the Varjo: https://mattsoson.com/cycle-7

[Nicolas Abondano]

I tried using ZED sdk for this effect but it didn't worked because the green screen prefab was designed for standalone. Do you have any suggestions on how to do it in Unity? I'm studied and haven't been able to do much progress

Edited August 29, 2023 by Nicolas Abondano

[Nicolas Abondano]

Also do you know if there's been any confirmation than on Quest 3 we will have able to the texture layer to do effects like chroma keying? I heard someone saying is possible but I couldn't find anything that confirms it

Edited August 29, 2023 by Nicolas Abondano