Accessing device camera feed while using hand tracking in Unity

[ericvids]

The Hand Tracking SDK requires the headset's back-facing camera to be enabled in SteamVR settings. However, this stops Unity from being able to access the same device camera as a WebCamTexture when I import the Unity plugin into my project.  Is there a workaround to get camera frames as textures, either through the plugin itself (since obviously the plugin initialized it) or some other means, perhaps my own Unity plugin?  (I'm ok with having to write C++ code, I just don't know what camera API to use to be compatible with the Hand Tracking SDK.)

Purpose: I want to simulate an Augmented Reality setup with hand tracking, mimicking a (very crude) HoloLens.  I'm using the original HTC Vive.

[zzy]

Hi @ericvids

From my previous experiences, I think cameras from SteamVR can be accessed by multiple callers. A quick example is that when hand tracking is running, you can still see the camera preview in camera tab from SteamVR settings.

As for how to access the camera frame/textures, you can refer to the OpenVR sample code here: https://github.com/ValveSoftware/openvr/tree/master/samples/tracked_camera_openvr_sample

I'll have a check next week to see if I can reproduce this on my machine.

[zzy]

Hi @ericvids

After a quick test, I cannot reproduce this problem locally. My steps are listed below. Please let me know if I missed something.

1. Unity version is 2018.4.19f, Vive Hand Tracking SDK 0.9.3, SteamVR plugin 2.6.1 (latest from Asset store). SteamVR version 1.14.16
2. I used SteamVR_TestTrackedCamera example scene as a start point, since this already has a camera texture. Please first make sure this scene alone can work fine.
3. Attach GestureProvider script to [CameraRig]/Camera game object.
4. For display purpose, add LeftHandRenderer and RightHandRenderer prefabs to the scene. Prefabs are inside Assets/ViveHandTracking/Prefab folder.
5. Play the scene now, you can see both hand skeleton and camera texture. Tested on both Vive and Vive Pro.

[ericvids]

Thank you for your help @zzy -- the whole process you outlined is flawless!

I was previously not using the SteamVR plugin (instead relying on Unity's built-in OpenVR support, which was rather limited), so I was trying to get Unity's own WebCamTexture to work for providing the camera feed. As it turns out, WebCamTexture had a bunch of problems (most important of which is that it exclusively locks the camera). Thanks for pointing me towards the SteamVR TrackedCamera API! @ericvids