openxr OpenXR does not report XR_EXT_eye_gaze_interaction nor XR_HTC_facial_tracking as supported

[DragonDreams]

I managed to get the problem with the SRanipal runtime not running (bad USB cable). Now I tried the SDK examples (FaceGym) to see if the runtime works. This example though crashes and quits which means something is wrong. I then started my DEMoCap (which is the reason for all this) since it supports the required OpenXR extensions XR_EXT_eye_gaze_interaction and XR_HTC_facial_tracking. This did not work either since the SteamVR using the SRanipal runtime does not report these two extensions as present:

II [OpenXR] [2023-04-10 19:35:15] Start Runtime
II [OpenXR] [2023-04-10 19:35:15] Create OpenXR Instance
II [OpenXR] [2023-04-10 19:35:15] Extensions:
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_vulkan_enable: 8
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_vulkan_enable2: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_D3D11_enable: 9
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_D3D12_enable: 9
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_opengl_enable: 10
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_win32_convert_performance_counter_time: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_win32_appcontainer_compatible: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_binding_modification: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_composition_layer_depth: 6
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_visibility_mask: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_active_action_set_priority: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_dpad_binding: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_frame_composition_report: 4
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hand_tracking: 4
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hand_joints_motion_range: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hp_mixed_reality_controller: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_palm_pose: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_FB_display_refresh_rate: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_HTC_vive_cosmos_controller_interaction: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_HTC_vive_focus3_controller_interaction: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_MND_headless: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_VALVE_analog_threshold: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_HTCX_vive_tracker_interaction: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_debug_utils: 4
II [OpenXR] [2023-04-10 19:35:15] Supported Extensions:
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_opengl_enable: 10
II [OpenXR] [2023-04-10 19:35:15] - XR_KHR_visibility_mask: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hand_joints_motion_range: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hand_tracking: 4
II [OpenXR] [2023-04-10 19:35:15] - XR_HTCX_vive_tracker_interaction: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_hp_mixed_reality_controller: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_HTC_vive_cosmos_controller_interaction: 1
II [OpenXR] [2023-04-10 19:35:15] - XR_HTC_vive_focus3_controller_interaction: 2
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_debug_utils: 4
II [OpenXR] [2023-04-10 19:35:15] Not upported Extensions:
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_eye_gaze_interaction
II [OpenXR] [2023-04-10 19:35:15] - XR_FB_passthrough
II [OpenXR] [2023-04-10 19:35:15] - XR_HTC_facial_tracking
II [OpenXR] [2023-04-10 19:35:15] - XR_EXT_samsung_odyssey_controller
II [OpenXR] [2023-04-10 19:35:15] - XR_HUAWEI_controller_interaction
II [OpenXR] [2023-04-10 19:35:15] - XR_MSFT_hand_interaction

Obviously without this extension being reported this all can not work.

The runtime reports this in the about dialog:

What is missing? Without this extension neither eye tracking nor face tracking is possible.

[DragonDreams]

Looks like VIVE Console on Steam is required to get the extension. But "VIVE Console" seems to be unstable and crashes. I launched into a debugger and found this:

The code in my application where the runtime crashes is this:

XrFacialTrackerCreateInfoHTC createInfo;
memset( &createInfo, 0, sizeof( createInfo ) );
createInfo.type = XR_TYPE_FACIAL_TRACKER_CREATE_INFO_HTC;
createInfo.facialTrackingType = XR_FACIAL_TRACKING_TYPE_EYE_DEFAULT_HTC;
instance.xrCreateFacialTrackerHTC( session.GetSession(), &createInfo, &pEyeTracker );

The instruction pointer of this thread is inside this function call. The crashing thread is the SR runtime as shown by the debugger here:

>	ucrtbase.dll!abort()	Unknown
 	ucrtbase.dll!terminate()	Unknown
 	vcruntime140.dll!FindHandler<__FrameHandler3>(EHExceptionRecord * pExcept, unsigned __int64 * pRN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC, const _s_FuncInfo * pFuncInfo, unsigned char recursive, int CatchDepth, unsigned __int64 * pMarkerRN) Line 776	C++
 	vcruntime140.dll!__InternalCxxFrameHandler<__FrameHandler3>(EHExceptionRecord * pExcept, unsigned __int64 * pRN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC, const _s_FuncInfo * pFuncInfo, int CatchDepth, unsigned __int64 * pMarkerRN, unsigned char recursive) Line 399	C++
 	vcruntime140.dll!__CxxFrameHandler3(EHExceptionRecord * pExcept, unsigned __int64 RN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC) Line 283	C++
 	ntdll.dll!RtlpExecuteHandlerForException()	Unknown
 	ntdll.dll!RtlDispatchException()	Unknown
 	ntdll.dll!KiUserExceptionDispatch()	Unknown
 	[External Code]	
 	ViveSR_Client.dll!00007fff19d5e24b()	Unknown
 	ViveSR_Client.dll!00007fff19d5d476()	Unknown
 	ViveSR_Client.dll!00007fff19d530f6()	Unknown
 	ViveSR_Client.dll!00007fff19d518d4()	Unknown
 	ViveSR_Client.dll!00007fff19d521e7()	Unknown
 	ViveSR_Client.dll!00007fff19d511a9()	Unknown
 	ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Why is ViveSR_Client.dll crashing? Is "VIVE Console" broken? What can I do to prevent this crash?

[Alex_HTC]

@DragonDreams
2 suggestions when we tend to see this:
1) Make sure you're using the latest version of sranipal runtime
2) make sure openxr provider is set correctly -- afaik it is in the registry as Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime but also exposed in steamvr settings

[DragonDreams]

I have completely removed SRanipal runtime and VIVE Console and reinstalled VIVE Console. The problem persists. I then completely removed all again and tried the VIVE Software download instead. This one seems to install and I can now enable the extensions XR_EXT_eye_gaze_interaction and XR_HTC_facial_tracking. There are some problems where I'm stuck.

1) XR_EXT_eye_gaze_interaction always reports no pose at all although I've done calibration and the calibration test (light up dots by looking at them) is working. I get no error yet the pose action as described in the OpenXR specification is receiving no input.

2) I'm using Index Controllers for finger tracking. The VIVE Pro Eye now for some reason does front camera finger tracking which sort of replaces/overlays the Index Controller finger tracking. This results in the fingers to look broken (bending in the wrong direction) and the hand location is high up in the sky. I have the suspicion that the SRanipal runtime sources both Index Controller and front camera hand tracking and sort of adds them together. I don't know what I can do against this.

[DragonDreams]

I did some more testing and the XR_HTC_facial_tracking also reports always 0 weights for anything. It is as if something is missing so my application gets no gaze nor facial information although the extensions are enabled.

I noticed there are VIVE_APILAYER_* layers reported. Are these required for the extensions to work properly? I can not find any information by searching for these layer names on the internet.

Right now nothing works at all which is frustrating since the documentation is 0 in that regard.

[DragonDreams]

@Alex_HTC I found some strange log entries. I don't think they are the reason for the problems but I still think they are strange:

II [OpenXR] [2023-04-13 16:04:11] Debug: xrGetSystemProperties: VUID-XrSystemHandTrackingPropertiesEXT-next-next - Invalid structure(s) in "next" chain for XrSystemHandTrackingPropertiesEXT struct "next"
II [OpenXR] [2023-04-13 16:04:11] Debug: xrGetSystemProperties: VUID-XrSystemProperties-next-next - Invalid structure(s) in "next" chain for XrSystemProperties struct "next"
II [OpenXR] [2023-04-13 16:04:11] Debug: xrGetSystemProperties: VUID-xrGetSystemProperties-properties-parameter - Command xrGetSystemProperties param properties is invalid

The code used is this: https://github.com/LordOfDragons/dragengine/blob/e9408aacbb5e546c80dcb3f25f4044327043bb2d/src/modules/vr/openxr/src/deoxrSystem.cpp#L59

With line 96 being the call leading to this log entry. I think the call is correct and I do get the positive result back stating the eye gaze and facial tracking is present. Could it be the validation layer has a bug here?

Other than that I get no messages from validation so why am I not getting any input neither from the eye tracker nor from the facial tracker?

[DragonDreams]

@Alex_HTC Is anybody doing support in here or what's this forum for?

[Alex_HTC]

@DragonDreams

Before you launch the app, what status is the sranipal runtime reporting https://dl.vive.com/Tracker/Guideline/Vive Face Tracker Developer Quick Start.pdf

Can you try the unity example and see if it reports the same issues? https://developer.vive.com/resources/openxr/openxr-mobile/tutorials/unity/getting-data-of-facial-tracking/ the goal of this is to further understand if there are any remaining hardware issues (like a usb 2.0 cable causing issues) and to further isolate the issue

[DragonDreams]

23 hours ago, Alex_HTC said:

@DragonDreams

Before you launch the app, what status is the sranipal runtime reporting https://dl.vive.com/Tracker/Guideline/Vive Face Tracker Developer Quick Start.pdf

Can you try the unity example and see if it reports the same issues? https://developer.vive.com/resources/openxr/openxr-mobile/tutorials/unity/getting-data-of-facial-tracking/ the goal of this is to further understand if there are any remaining hardware issues (like a usb 2.0 cable causing issues) and to further isolate the issue

The robot symbol has green eyes and green mouth. The eye calibration tool test (light dots by looking at them) is working.

With the Unity example at startup the logo head is visible but then it crashes (or just shuts down) after some time. The C example reports error to not be able to create tracking engine (timeout or something like this).

The USB cable I did change already. It's a USB 3 cable on a USB 3 port. This one is working fine and can be ruled out as problem source.

The problem is in my test I get for gaze xrLocateSpace returns all 0 (especially flags valid bit and tracking bit not set) and for facial tracking  xrGetFacialExpressionsHTC returns all 0. Could it be the VR runtime does not grant my app (and the example apps) access to the hardware although the SRanipal did successfully start up. Any ideas?

[Alex_HTC]

@DragonDreams You're getting 0 for all values, if i understand correctly on the native side.

To me this sounds like a configuration or hardware failure. 

When running the unity example - does the face move at all? Or do you get that far?

The native issue may be a red herring if it's compiling against older sranipal dlls as there was a required update relatively recently, which is why I'm asking more about the unity side since that is in a known-good state.

I'll inquire further internally.