Jump to content

Vive Pro front camera algorithm for see-through mode


Recommended Posts


I was wondering if anyone could share what is the algorithm or worflow used by SR Works for example to make a see-through mode ?

So first, is there any possibilities to get each camera intrinsics from OpenVR ? Do I need to call another API for that ? 

In OpenVR I am able to acquire a current front facing camera framebuffer, which is upside down and displayed as top/bottom for the left and right camera in the same image ( i actually use the distorted framebuffer).  The problem comes after that... what can I do with this image ?

In Unity, i apply the image on a mesh material as a texture. This is a particular mesh : it's an anamorphic plane (96° horizontally, 80° vertically). this mesh is transformed by the HMD matrices, and it's offseted a little on forward axis by few centimeters. Then i scaled up the mesh to become really far without changing anything in the field of view. 

If test it this way, everything seems promising, but if i add 3d objects in the scene, we can clearly see this is not working... The rate of rotation/translation is inequal.


Any idea ?


Thank you.


@Daniel_Y @reneeclchen @Andy.YC_Wang @Jad

Link to comment
Share on other sites

Thank you for the tip @Daniel_Y, I wasn't sure the method gives the right informations according to this issue : https://github.com/ValveSoftware/openvr/issues/1100

For now, when I acquire the TrackedCamera framebuffer, I simply use it as a texture on a shader applied on a surface

using UnityEngine;
using Valve.VR;

public class SteamVR_CameraBackground : Monobehaviour {
    public MeshRenderer TargetRenderer;
    public bool Undistored = false;
    Texture2D m_videoTex;

    private void OnEnable()
        SteamVR_TrackedCamera.VideoStreamTexture videoSource = SteamVR_TrackedCamera.Source(Undistored);

        if (!videoSource.hasCamera)
            enabled = false;


    private void OnDisable()
         SteamVR_TrackedCamera.VideoStreamTexture videoSource = SteamVR_TrackedCamera.Source(Undistored);

    private void Update()
        SteamVR_TrackedCamera.VideoStreamTexture videoSource = SteamVR_TrackedCamera.Source(Undistored);
        m_videoTex = videoSource.texture;
        TargetRenderer.material.mainTexture = m_videoTex;

The texture layout looks like this: 


As I work on single pass stereo mode, Here's how I re-project the image:  

Shader "Unlit/StereoDistort" {
        _MainTex("Texture", 2D) = "white" {}

        Tags { "RenderType" = "Opaque" }

        Pass {
                #pragma vertex vert
                #pragma fragment frag

                #include "UnityCG.cginc"

                struct appdata_t {
                    float4 vertex : POSITION;
                    float2 uv : TEXCOORD0;

                struct v2f {
                    float4 vertex : SV_POSITION;
                    float2 uv : TEXCOORD0;

                uniform sampler2D _MainTex;
                uniform float4 _MainTex_ST;

                v2f vert(appdata_t v)
                    v2f o;
                    UNITY_INITIALIZE_OUTPUT(v2f, o);
                    o.vertex = UnityObjectToClipPos(v.vertex);
                    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
                    return o;

                fixed4 frag(v2f i) : SV_Target

                    float offset = lerp(0.5, 0, unity_StereoEyeIndex); // switch 0.5 and 0 values to switch between left and right eyes
                    float vCoord = offset + (1. - i.uv.y) * .5;
                    fixed4 col = tex2D(_MainTex, float2(i.uv.x, vCoord));
                    return col;



In your VIveSR Unity package, you use 2 surfaces, on for each eye, when I use only 1 surface. I don't feel an offset problem between eyes though.

The main issue still the inequality when I rotate my head or when i move forward, the 3D scene doesn't stay at the same place according to the Camera framebuffer. I am pretty sure I miss a step to make it work, but I can't figure out what.

Thank you again.


Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...