VIVE Eye Tracking at 120hz

[Corvus]

VIVE Eye Tracking at 120hz

---

 

Note: First setup the VIVE Eye Tracking SDK and verify basic functionality (calibration & demo scene)

Note: Callback runs on a separate thread to report data at ~120hz

Note: Unity is not thread-safe and cannot call any UnityEngine api from within callback thread.

 

Unity Guide

---

 

Setup Project for Eye Tracking

• Import VIVE Eye Tracking SDK Unity Plugin (SRanipal)

• Add the SRanipal Eye Framework to the Unity scene

  • Drag "SRanipal Eye Framework" prefab into scene hierarchy

    or

  • Attach "SRanipal_Eye_Framework.cs" script to gameobject in scene

• Setup Framework Settings

  • Check "Enable Eye" (enabled by default)

  • Check "Enable Eye Data Callback"

 

 

Setup EyeData Callback

 

Register EyeData Callback

Note: Only register callback when the eye tracking framework status is working and do not register more than one callback at once.

SRanipal_Eye.WrapperRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));

 

Use EyeData Callback

        private static EyeData eyeData = new EyeData();

        /// Required class for IL2CPP scripting backend support
        internal class MonoPInvokeCallbackAttribute : System.Attribute
        {
            public MonoPInvokeCallbackAttribute() { }
        }

        [MonoPInvokeCallback]
        private static void EyeCallback(ref EyeData eye_data)
        {
            eyeData = eye_data;
            // do stuff with eyeData...
        }

 

Unregister EyeData Callback

Note: Must unregister callback when complete or application is disabled/quit.

                SRanipal_Eye.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));

 

Unity Sample Code

---

using UnityEngine;
using ViveSR.anipal.Eye;
using System.Runtime.InteropServices;

/// <summary>
/// Example usage for eye tracking callback
/// Note: Callback runs on a separate thread to report at ~120hz.
/// Unity is not threadsafe and cannot call any UnityEngine api from within callback thread.
/// </summary>
public class CallbackExample : MonoBehaviour
{
    private static EyeData eyeData = new EyeData();
    private static bool eye_callback_registered = false;

    private void Update()
    {
        if (SRanipal_Eye_Framework.Status != SRanipal_Eye_Framework.FrameworkStatus.WORKING) return;

        if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == true && eye_callback_registered == false)
        {
            SRanipal_Eye.WrapperRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = true;
        }
        else if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == false && eye_callback_registered == true)
        {
            SRanipal_Eye.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }
    }

    private void OnDisable()
    {
        Release();
    }

    void OnApplicationQuit()
    {
        Release();
    }

    /// <summary>
    /// Release callback thread when disabled or quit
    /// </summary>
    private static void Release()
    {
        if (eye_callback_registered == true)
        {
            SRanipal_Eye.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }
    }

    /// <summary>
    /// Required class for IL2CPP scripting backend support
    /// </summary>
    internal class MonoPInvokeCallbackAttribute : System.Attribute
    {
        public MonoPInvokeCallbackAttribute() { }
    }

    /// <summary>
    /// Eye tracking data callback thread.
    /// Reports data at ~120hz
    /// MonoPInvokeCallback attribute required for IL2CPP scripting backend
    /// </summary>
    /// <param name="eye_data">Reference to latest eye_data</param>
    [MonoPInvokeCallback]
    private static void EyeCallback(ref EyeData eye_data)
    {
        eyeData = eye_data;
        // do stuff with eyeData...
    }
}

[jboss]

@Corvus Thanks for this guide. I'll give it a try after the weekend. One question: in the screenshot you use Eye Version 1. Does that matter? Or does it work with both version 1 and 2?

[Corvus]

v2 added more face blend morphs (EyeExpression  expression_data) but otherwise SHOULD be the same. There was a report recently of v2 eyedata timestamp reporting differently than v1 timestamp, if so that would be a bug and will try to get fixed in next release.

[apellisscot]

Thanks @Corvus

I followed your instructions and could still not reach the 120 Hz (instead, it is acquired at ~62 Hz). I tried v1 and v2 and only had a static counter incrementing in the callback function. Could it be that it is limited by our HW? We have following specs:

• Windows 10Pro
• Intel i7-10750H (specs can be found here)
• 32GB Ram
• GeForce RTX 2070 with Max-Q Design

Following tool versions are used:

• SRanipal SDK & Runtime 1.3.1.1
• Unity 2019.4.18f1

Am I right in the assumption that the key limiting factor is the CPU and not the GPU? We will soon have a better performing machine to try it out.

PS: If you need a code snippet, please see refer to this thread. Thx!

[Corvus]

@apellisscot I have gotten 120hz on similar specs but will try to verify/send a build for you to test. Also, can you share any info about your "EyeMeasureDataWriter"? If it's running on the Unity main thread it will be limited to the update loop hz.

[Corvus]

@apellisscot Here are my specs and I'm getting 8/9ms updates (~120hz). DM me if you would like a build to test/verify with.

• Windows 10
• Intel i7-9750H
• 16GB Ram
• RTX 2070 Max-Q

 

• Unity 2019.4.17f1

 

using UnityEngine;
using ViveSR.anipal.Eye;
using System.Runtime.InteropServices;
using UnityEngine.UI;

/// <summary>
/// Example usage for eye tracking callback
/// Note: Callback runs on a separate thread to report at ~120hz.
/// Unity is not threadsafe and cannot call any UnityEngine api from within callback thread.
/// </summary>
public class test120hz : MonoBehaviour
{
    private static EyeData eyeData = new EyeData();
    private static bool eye_callback_registered = false;

    public Text uiText;
    private float updateSpeed = 0;
    private static float lastTime, currentTime;


    void Update()
    {
        if (SRanipal_Eye_Framework.Status != SRanipal_Eye_Framework.FrameworkStatus.WORKING) return;


        if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == true && eye_callback_registered == false)
        {
            SRanipal_Eye.WrapperRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = true;
        }
        else if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == false && eye_callback_registered == true)
        {
            SRanipal_Eye.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }

        updateSpeed = currentTime - lastTime;
        uiText.text = updateSpeed.ToString() + " ms";
    }

    private void OnDisable()
    {
        Release();
    }

    void OnApplicationQuit()
    {
        Release();
    }

    /// <summary>
    /// Release callback thread when disabled or quit
    /// </summary>
    private static void Release()
    {
        if (eye_callback_registered == true)
        {
            SRanipal_Eye.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }
    }

    /// <summary>
    /// Required class for IL2CPP scripting backend support
    /// </summary>
    internal class MonoPInvokeCallbackAttribute : System.Attribute
    {
        public MonoPInvokeCallbackAttribute() { }
    }

    /// <summary>
    /// Eye tracking data callback thread.
    /// Reports data at ~120hz
    /// MonoPInvokeCallback attribute required for IL2CPP scripting backend
    /// </summary>
    /// <param name="eye_data">Reference to latest eye_data</param>
    [MonoPInvokeCallback]
    private static void EyeCallback(ref EyeData eye_data)
    {
        eyeData = eye_data;
        // do stuff with eyeData...

        lastTime = currentTime;
        currentTime = eyeData.timestamp;
    }
}

 

[apellisscot]

Thanks @Corvus, I had removed the call to "EyeMeasureDataWriter" and placed a static counter there - so similarly as you do. The only difference I see is that I register the callback in the Monobehaviour's "Start()" method and you do it in the "Update()" method. Although I do not see any reason why this should change something, I will give it a try next week.

It would be great if you could send me the build. I've sent you a DM.

Edited February 13, 2021 by apellisscot

[Corvus]

@apellisscot Demo attached (rename .zip).

Eye Tracking 120hz Test.zip_

[apellisscot]

Thanks @Corvus, it still captures data at a sampling frequency of around 60 Hz (see video attached). Sometimes the period drops to 8 ms, but this occurs rarely. Are there any CPU configurations to perform? I tried with deactivated hyper-threading, but did not have any luck.

Soon we will have a higher performing PC to test with. I will let you know the results there.

[HackPerception]

@apellisscot - It might be a PC hardware bottleneck. What system specs are you testing on?

On decently spec'ed PC's we'd expect that project to run at/near a steady 8ms.