How to integrate sranipal c api in C++ rendering project?

[VIVE_chengnay]

Hi @vinciDa,

I found old topic did mention the differences between V1 and V2.

I only use V2 for my development and I suggest to use V2 for your Unreal development.

[vinciDa]

@VIVE_chengnay, but one thing I have noticed, integrating eye tracking in the code-base (VRWorks' Variable Rate Shading) is dramatically dropping the frame rate. For instance, the `mouse cursor tracking = 60 fps (avg)` whereas when using the `gaze tracking = 7 fps (avg)`. Will it be call normal, or I am doing something wrong? I need the 90 fps

Edited June 13 by vinciDa

[VIVE_chengnay]

Hi @vinciDa,

Sounds like you are getting eye data now?

7 fps? There must be something wrong with it.

[vinciDa]

Hi @VIVE_chengnay, I'm back to the problem now. After some re-evaluation, I am now getting an average frame rate of around 32 fps, sometimes 32.99, but never 33 or higher. Without eye tracking, the average frame rate is 90 fps or slightly higher if I even use a mouse cursor as the tracker. So there are currently two challenges (Using Vive Pro Eye, on Windows 10, NVidia VRWorks code base, OpenVR):

• Enabling eye tracking lowers the frame rate (1/3).  How to resolve this problem? I am sure the eye tracker has a 120 Hz frequency. So, How do achieve that? Some random googling suggests it may be an `asynchronous` problem. Another random google even suggest to use `  SR_ANIPAL int RegisterEyeDataCallback(VivesrEyeDataCallback callback);` under `SRanipal_Eye.h`, but do not know how to use it.

Here is what I have tried, and looking for your suggestions:

#include "include/SRanipal.h"
#include "include/SRanipal_Eye.h"
#pragma comment (lib, "SRanipal.lib")
using namespace ViveSR;
ViveSR::anipal::Eye::EyeData eye_data; // as global variable
...

bool VRWorksSample::Init(HINSTANCE hInstance) // vrworks method
{
...
   ViveSR::anipal::Initial(ViveSR::anipal::Eye::ANIPAL_TYPE_EYE, NULL);
   return true;
}

...
void VRWorksSample::OnRender() // main rendering loop
{
if (g_foveatedRenderingEnabled)
	{
		float GazeNormX = 0.0f;
		float GazeNormY = 0.0f;
...
			if (g_pNvFoveatedRenderingHelper->m_eyeTrackingMode == EYE_TRACKING_MODE_SRANIPAL)
			{
			ViveSR::anipal::Eye::GetEyeData(&eye_data);
			GazeNormX = eye_data.verbose_data.combined.eye_data.gaze_direction_normalized.x;
			GazeNormY = eye_data.verbose_data.combined.eye_data.gaze_direction_normalized.y;
            }
        // vrworks side
     	g_pNvFoveatedRenderingHelper->UpdateGazeData(GazeNormX, GazeNormY, IsVROrFakeVRActive());
		g_pNvFoveatedRenderingHelper->LatchGazeData();

g_gazeFinalLeft = util::makefloat2(g_pNvFoveatedRenderingHelper->m_gazeDataParams.sStereoData.sLeftEye.fGazeNormalizedLocation[0], g_pNvFoveatedRenderingHelper->m_gazeDataParams.sStereoData.sLeftEye.fGazeNormalizedLocation[1]);

		g_gazeFinalRight = util::makefloat2(g_pNvFoveatedRenderingHelper->m_gazeDataParams.sStereoData.sRightEye.fGazeNormalizedLocation[0], g_pNvFoveatedRenderingHelper->m_gazeDataParams.sStereoData.sRightEye.fGazeNormalizedLocation[1]);
}

 

[VIVE_chengnay]

Hi @vinciDa,

SRanipal was legacy SDK and we encourage all developers move to using OpenXR.

Here is the tutorial of OpenXR eye gaze for your reference.

https://developer.vive.com/resources/openxr/unreal/unreal-tutorials/eyegaze/

Is this OK for you?

 

[vinciDa]

This is not okay for me. For a research project, it is too much to use a game engine like UE or Unity. Also, the plugin works for UE 5.0 or latest.

Back to my problem, (1) how can I get a 120 Hz tracking frequency and (2) that should not block my actual frame rate? I am using DirectX11, OpenVR.

[VIVE_chengnay]

Hi @vinciDa,

If you don't check g_foveatedRenderingEnabled and will the fps goes up?

On 7/10/2024 at 11:59 PM, vinciDa said:

Enabling eye tracking lowers the frame rate (1/3).  How to resolve this problem?

Do you mean everything in content frame rate drops after you enable eyetracking?

Are you able to provide sample project for further investigation?

[vinciDa]

The g_foveatedRenderingEnabled has nothing to do with the frame rate drop because, besides sranipal, I have also cursor-based tracking, which is working fine. The good news that, I have just solved the frame drop problem with the C++ thread.

One more question, I am doing the calibration with steamvr. So, while I am running the sranipal runtime, I guess that is sharing the calibrated eye data. Am I right? Or, do I need to add calibration in the code as well?

 

[VIVE_chengnay]

Hi @vinciDa,

I don't think you need to add calibration in the code.

Hope you can find your answer in below link.