[VBS & PCVR] - How to use Hand tracking

[Vivi Wu]

PC VR contents could now enjoy Focus 3 hand tracking via OpenVR Skeleton Input API or OpenXR hand tracking extension. No additional porting effort is required.

VBS leverages Focus 3 which easily switch between controller and hand interaction by simply pick up or put down controllers.

VBS also supports pinch gesture as controller trigger button, so that you could play various VR contents with just hands. Such as The Lab: Xortex.

 

VBS hand tracking for developers

There are two open SDKs that VBS supports to bring Focus 3 hand tracking to PC contents. Developers can adopt open standard SDK to implement his hand tracking contents without being locked in proprietary hand tracking SDK.

·        OpenVR Skeleton Input

https://github.com/ValveSoftware/openvr/wiki/SteamVR-Skeletal-Input

·        OpenXR hand tracking

https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#XR_EXT_hand_tracking

 

OpenXR hand tracking Tutorial:

·        Unity: https://developer.vive.com/resources/openxr/openxr-pcvr/tutorials/unity/how-integrate-hand-tracking-data-your-hand-model/

·        Unreal: https://developer.vive.com/resources/openxr/openxr-pcvr/tutorials/unreal-engine/integrate-hand-tracking-data-your-hand-model/

 

To enable/disable nature hand tracking data from AIO headset:

Go to VBS > Settings > Preferences > Tracking

1. ☑️Hand tracking
   Nature hand tracking enabled.
   VBS will emulate a controller for each hand to SteamVR and translate pinch gesture to trigger event. Therefore, contents which don’t support hand tracking can still work.
   Simple put down controller to switch to nature hand, pick up controller to switch back to controller mode.
    
2. 🟦Hand tracking (Uncheck)
   Suppress nature hand tracking data
   VBS will still emulate hand poses (eletric hand) based on controller input status. Therefore, contents requires hand tracking can still work but not having nature hand poses.
    
3. ☑️Hand tracking
   ☑️Hand tracking and controller at the same time
   With Focus 3 June FOTA update, VBS (1.08) supports nature hand tracking and controller tracking at the same time. Hence, you can use one controller and one nature hand at the same time for particular VR experience.
    
4. ☑️Hand tracking
   ☑️Use VIVE Wrist Tracker for hand tracking
   VIVE Wrist Tracker will be not be a tracked device to SteamVR, instead it will be used for enhanced hand tracking. Therefore, you won’t see VIVE Wrist Tracker device in SteamVR.
    
5. ☑️Hand tracking
   🟦Use VIVE Wrist Tracker for hand tracking(Uncheck)
   VIVE Wrist Tracker is used as a tracked device, and VBS will emulate as VIVE Tracker to SteamVR. You will see device icon on SteamVR monitor.

 

More detail:

VBS for Developers - hand tracking.pdf

 

VBS for Developers - VIVE Wrist Tracker_V2.pdf

 

[mikeViewPort]

Thanks for the information @Vivi Wu. Is there any information/documentation on Gesture recognition for OpenXR / VBS?

[Vivi Wu]

Hi @mikeViewPort

Khronos don't have the specification for OpenXR hand gesture yet.

If there's any news, we'll publish it on VIVE website and VIVE Forum.

Thanks.

[mikeViewPort]

@Vivi Wu thanks for the reply. I know Microsoft has extensions on OpenXR with MRTK for the Hololens to allow gesture recognition while they waiting on it. Any plans that you might do the same? Hand tracking without gesture recognition isn't very... handy

[ForensicDJS]

I am trying to get "Get Pinch Active" to fire from Open XR Hand Interaction in Unreal Engine but without any luck. Not sure what I am missing. Also, InputAction TriggerRight/TriggerLeft works when pinching or making a fist but this only works on Windows, it doesn't get detected when launched on HTC Vive Focus 3/Android. Need help for a time sensitive project. 

[Vivi Wu]

Hi @ForensicDJS,

If you're an Android developer. Please refer AIO (Android) / OpenXR Hand Tracking & Wrist Tracker Tutorials

The tutorial for Hand Tracking  on Unreal Engine (Android) - Please refer Here