Eyetracking data not logging

[eugie]

Hello,

I wrote a data logging script referencing the script written by Yu Imaoka and Andri Flury at D-HEST, ETH Zurich. (https://doi.org/10.1016/j.visres.2013.02.007)

However, the data would not log and only the string from Data_txt() is logged onto the desired .txt file. 

I've been working on this for the last few weeks but I am lost in what the problem is. 

The unity project is a simple 360 video projection on a sphere, which runs smoothly, so I do not think it is the project problem. 

I would really appreciate the help. Thank you. 

 

Data output : 

 

 

Code : 

using System.Collections;
using System.Runtime.InteropServices;
using UnityEngine;
using System;
using System.IO;
using ViveSR.anipal.Eye;
using ViveSR.anipal;
using ViveSR;




public class Eyetracking : MonoBehaviour
{
    // ********************************************************************************************************************
    //
    //  Define user ID information.
    //  - The developers can define the user ID format such as "ABC_001". The ID is used for the name of text file 
    //    that records the measured eye movement data.
    // 
    // ********************************************************************************************************************
    public static string UserID = "02";       // Definte ID number such as 001, ABC001, etc.
    public static string Path = Directory.GetCurrentDirectory();
    string File_Path = Directory.GetCurrentDirectory() + "\\video_" + UserID + ".txt";


    // ********************************************************************************************************************
    //
    //  Parameters for time-related information.
    //
    // ********************************************************************************************************************
    public static int cnt_callback = 0;
    public int cnt_saccade = 0, Endbuffer = 3, SaccadeTimer = 30;
    float Timeout = 1.0f, InitialTimer = 0.0f;
    private static long SaccadeEndTime = 0;
    private static long MeasureTime, CurrentTime, MeasureEndTime = 0;
    private static float time_stamp;
    private static int frame;

    // ********************************************************************************************************************
    //
    //  Parameters for eye data.
    //
    // ********************************************************************************************************************
    private static EyeData_v2 eyeData = new EyeData_v2();
    public EyeParameter eye_parameter = new EyeParameter();
    public GazeRayParameter gaze = new GazeRayParameter();
    private static bool eye_callback_registered = false;
    private static UInt64 eye_valid_L, eye_valid_R;                 // The bits explaining the validity of eye data.
    private static float openness_L, openness_R;                    // The level of eye openness.
    private static float pupil_diameter_L, pupil_diameter_R;        // Diameter of pupil dilation.
    private static Vector2 pos_sensor_L, pos_sensor_R;              // Positions of pupils.
    private static Vector3 gaze_origin_L, gaze_origin_R;            // Position of gaze origin.
    private static Vector3 gaze_direct_L, gaze_direct_R;            // Direction of gaze ray.
    private static float frown_L, frown_R;                          // The level of user's frown.
    private static float squeeze_L, squeeze_R;                      // The level to show how the eye is closed tightly.
    private static float wide_L, wide_R;                            // The level to show how the eye is open widely.
    private static double gaze_sensitive;                           // The sensitive factor of gaze ray.
    private static float distance_C;                                // Distance from the central point of right and left eyes.
    private static bool distance_valid_C;                           // Validity of combined data of right and left eyes.
    public bool cal_need;                                           // Calibration judge.
    public bool result_cal;                                         // Result of calibration.
    private static int track_imp_cnt = 0;
    private static TrackingImprovement[] track_imp_item;



    // ********************************************************************************************************************
    //
    //  Start is called before the first frame update. The Start() function is performed only one time.
    //
    // ********************************************************************************************************************
    void Start()
    {
        InputUserID();                              // Check if the file with the same ID exists.
        Invoke("SystemCheck", 0.5f);                // System check.
        //SRanipal_Eye_v2.LaunchEyeCalibration();     // Perform calibration for eye tracking.
        //Calibration();
        //TargetPosition();                           // Implement the targets on the VR view.
        Invoke("Measurement", 0.5f);                // Start the measurement of ocular movements in a separate callback function.  
    }



    // ********************************************************************************************************************
    //
    //  Checks if the filename with the same user ID already exists. If so, you need to change the name of UserID.
    //
    // ********************************************************************************************************************
    void InputUserID()
    {
        Debug.Log(File_Path);

        if (File.Exists(File_Path))
        {
            Debug.Log("File with the same UserID already exists. Please change the UserID in the C# code.");

            //  When the same file name is found, we stop playing Unity.

            if (UnityEditor.EditorApplication.isPlaying)
            {
                UnityEditor.EditorApplication.isPlaying = false;
            }
        }
    }



    // ********************************************************************************************************************
    //
    //  Check if the system works properly.
    //
    // ********************************************************************************************************************
    void SystemCheck()
    {
        if (SRanipal_Eye_API.GetEyeData_v2(ref eyeData) == ViveSR.Error.WORK)
        {
            Debug.Log("Device is working properly.");
        }

        if (SRanipal_Eye_API.GetEyeParameter(ref eye_parameter) == ViveSR.Error.WORK)
        {
            Debug.Log("Eye parameters are measured.");
        }

        //  Check again if the initialisation of eye tracking functions successfully. If not, we stop playing Unity.
        Error result_eye_init = SRanipal_API.Initial(SRanipal_Eye_v2.ANIPAL_TYPE_EYE_V2, IntPtr.Zero);

        if (result_eye_init == Error.WORK)
        {
            Debug.Log("[SRanipal] Initial Eye v2: " + result_eye_init);
        }
        else
        {
            Debug.LogError("[SRanipal] Initial Eye v2: " + result_eye_init);

            if (UnityEditor.EditorApplication.isPlaying)
            {
                UnityEditor.EditorApplication.isPlaying = false;    // Stops Unity editor.
            }
        }
    }



    // ********************************************************************************************************************
    //
    //  Calibration is performed if the calibration is necessary.
    //
    // ********************************************************************************************************************
    void Calibration()
    {
        SRanipal_Eye_API.IsUserNeedCalibration(ref cal_need);           // Check the calibration status. If needed, we perform the calibration.

        if (cal_need == true)
        {
            result_cal = SRanipal_Eye_v2.LaunchEyeCalibration();

            if (result_cal == true)
            {
                Debug.Log("Calibration is done successfully.");
            }

            else
            {
                Debug.Log("Calibration is failed.");
                if (UnityEditor.EditorApplication.isPlaying)
                {
                    UnityEditor.EditorApplication.isPlaying = false;    // Stops Unity editor if the calibration if failed.
                }
            }
        }

        if (cal_need == false)
        {
            Debug.Log("Calibration is not necessary");
        }
    }





    // ********************************************************************************************************************
    //
    //  Create a text file and header names of each column to store the measured data of eye movements.
    //
    // ********************************************************************************************************************
    void Data_txt()
    {
        string variable =
        "time(100ns)" + "," +
        "time_stamp(ms)" + "," +
        "frame" + "," +
        "eye_valid_L" + "," +
        "eye_valid_R" + "," +
        "openness_L" + "," +
        "openness_R" + "," +
        "pupil_diameter_L(mm)" + "," +
        "pupil_diameter_R(mm)" + "," +
        "pos_sensor_L.x" + "," +
        "pos_sensor_L.y" + "," +
        "pos_sensor_R.x" + "," +
        "pos_sensor_R.y" + "," +
        "gaze_origin_L.x(mm)" + "," +
        "gaze_origin_L.y(mm)" + "," +
        "gaze_origin_L.z(mm)" + "," +
        "gaze_origin_R.x(mm)" + "," +
        "gaze_origin_R.y(mm)" + "," +
        "gaze_origin_R.z(mm)" + "," +
        "gaze_direct_L.x" + "," +
        "gaze_direct_L.y" + "," +
        "gaze_direct_L.z" + "," +
        "gaze_direct_R.x" + "," +
        "gaze_direct_R.y" + "," +
        "gaze_direct_R.z" + "," +
        "gaze_sensitive" + "," +
        "frown_L" + "," +
        "frown_R" + "," +
        "squeeze_L" + "," +
        "squeeze_R" + "," +
        "wide_L" + "," +
        "wide_R" + "," +
        "distance_valid_C" + "," +
        "distance_C(mm)" + "," +
        "track_imp_cnt" +
        Environment.NewLine;

        File.AppendAllText("video_" + UserID + ".txt", variable);
    }



    // ********************************************************************************************************************
    //
    //  Measure eye movements in a callback function that HTC SRanipal provides.
    //
    // ********************************************************************************************************************
    void Measurement()
    {
        EyeParameter eye_parameter = new EyeParameter();
        SRanipal_Eye_API.GetEyeParameter(ref eye_parameter);
        Data_txt();

        if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == true && eye_callback_registered == false)
        {
            SRanipal_Eye_v2.WrapperRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye_v2.CallbackBasic)EyeCallback));
            eye_callback_registered = true;
        }

        else if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == false && eye_callback_registered == true)
        {
            SRanipal_Eye_v2.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye_v2.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }
    }



    // ********************************************************************************************************************
    //
    //  Callback function to record the eye movement data.
    //  Note that SRanipal_Eye_v2 does not work in the function below. It only works under UnityEngine.
    //
    // ********************************************************************************************************************
    private static void EyeCallback(ref EyeData_v2 eye_data)
    {
        EyeParameter eye_parameter = new EyeParameter();
        SRanipal_Eye_API.GetEyeParameter(ref eye_parameter);
        eyeData = eye_data;

        // ----------------------------------------------------------------------------------------------------------------
        //  Measure eye movements at the frequency of 120Hz until framecount reaches the maxframe count set.
        // ----------------------------------------------------------------------------------------------------------------
        ViveSR.Error error = SRanipal_Eye_API.GetEyeData_v2(ref eyeData);

        if (error == ViveSR.Error.WORK)
        {
            // --------------------------------------------------------------------------------------------------------
            //  Measure each parameter of eye data that are specified in the guideline of SRanipal SDK.
            // --------------------------------------------------------------------------------------------------------
            MeasureTime = DateTime.Now.Ticks;
            time_stamp = eyeData.timestamp;
            frame = eyeData.frame_sequence;
            eye_valid_L = eyeData.verbose_data.left.eye_data_validata_bit_mask;
            eye_valid_R = eyeData.verbose_data.right.eye_data_validata_bit_mask;
            openness_L = eyeData.verbose_data.left.eye_openness;
            openness_R = eyeData.verbose_data.right.eye_openness;
            pupil_diameter_L = eyeData.verbose_data.left.pupil_diameter_mm;
            pupil_diameter_R = eyeData.verbose_data.right.pupil_diameter_mm;
            pos_sensor_L = eyeData.verbose_data.left.pupil_position_in_sensor_area;
            pos_sensor_R = eyeData.verbose_data.right.pupil_position_in_sensor_area;
            gaze_origin_L = eyeData.verbose_data.left.gaze_origin_mm;
            gaze_origin_R = eyeData.verbose_data.right.gaze_origin_mm;
            gaze_direct_L = eyeData.verbose_data.left.gaze_direction_normalized;
            gaze_direct_R = eyeData.verbose_data.right.gaze_direction_normalized;
            gaze_sensitive = eye_parameter.gaze_ray_parameter.sensitive_factor;
            frown_L = eyeData.expression_data.left.eye_frown;
            frown_R = eyeData.expression_data.right.eye_frown;
            squeeze_L = eyeData.expression_data.left.eye_squeeze;
            squeeze_R = eyeData.expression_data.right.eye_squeeze;
            wide_L = eyeData.expression_data.left.eye_wide;
            wide_R = eyeData.expression_data.right.eye_wide;
            distance_valid_C = eyeData.verbose_data.combined.convergence_distance_validity;
            distance_C = eyeData.verbose_data.combined.convergence_distance_mm;
            track_imp_cnt = eyeData.verbose_data.tracking_improvements.count;
            ////track_imp_item = eyeData.verbose_data.tracking_improvements.items;

            //  Convert the measured data to string data to write in a text file.
            string value =
                MeasureTime.ToString() + "," +
                time_stamp.ToString() + "," +
                frame.ToString() + "," +
                eye_valid_L.ToString() + "," +
                eye_valid_R.ToString() + "," +
                openness_L.ToString() + "," +
                openness_R.ToString() + "," +
                pupil_diameter_L.ToString() + "," +
                pupil_diameter_R.ToString() + "," +
                pos_sensor_L.x.ToString() + "," +
                pos_sensor_L.y.ToString() + "," +
                pos_sensor_R.x.ToString() + "," +
                pos_sensor_R.y.ToString() + "," +
                gaze_origin_L.x.ToString() + "," +
                gaze_origin_L.y.ToString() + "," +
                gaze_origin_L.z.ToString() + "," +
                gaze_origin_R.x.ToString() + "," +
                gaze_origin_R.y.ToString() + "," +
                gaze_origin_R.z.ToString() + "," +
                gaze_direct_L.x.ToString() + "," +
                gaze_direct_L.y.ToString() + "," +
                gaze_direct_L.z.ToString() + "," +
                gaze_direct_R.x.ToString() + "," +
                gaze_direct_R.y.ToString() + "," +
                gaze_direct_R.z.ToString() + "," +
                gaze_sensitive.ToString() + "," +
                frown_L.ToString() + "," +
                frown_R.ToString() + "," +
                squeeze_L.ToString() + "," +
                squeeze_R.ToString() + "," +
                wide_L.ToString() + "," +
                wide_R.ToString() + "," +
                distance_valid_C.ToString() + "," +
                distance_C.ToString() + "," +
                track_imp_cnt.ToString() +
                //track_imp_item.ToString() +
                Environment.NewLine;

            File.AppendAllText("video_" + UserID + ".txt", value);

            cnt_callback++;
        }
    }
}

 

Edited April 22, 2021 by eugie
new image

[bfalandays]

Hi @eugie,

I edited your script slightly, combining it with the script posted by @Corvus in this other thread: VIVE Eye Tracking at 120hz - VIVE Eye and Facial Tracking SDK - Community Forum

I will post it below. But it is working fine for me, so thanks very much. I noticed that it doesn't work in other projects that are also using the Tobii XR SDK, perhaps that could be your problem?

 

EyeTrackingRecordData.cs

[eugie]

@bfalandays 

you were right! The tracking works fine now after I erased the Tobii XR SDK!! Thank you so much. 

[luiluks]

Hello,

first of all thanks very much @eugie for sharing your script. I'm also working on getting eye tracking data in a simple scene and the script was very helpful to get a sense of the sranipals functionality as a total unity and c# newbie. Unfortunately, even when trying your script with just the sranipals provided examples I'm stuck with just the string getting logged in the resulting file like, although no tobii packages are loaded. Do you have any further ideas as to what could be the problem? Or did you remove the tobii sdk not just from your project but from your whole machine?

Also @bfalandays, could you maybe dm me or reup your script so I could try with that version also?

Thanks very much! Really any help is appreciated!

[eugie]

@luiluks

No need to thank me 🙂 it is mostly from the the script written by Yu Imaoka and Andri Flury at D-HEST, ETH Zurich. (https://doi.org/10.1016/j.visres.2013.02.007) be sure to reference them if you use the script! 

did you update the file name every time? if the script detects a file with the same name, it would shut off. 

Do you have SRanipal Eye Framework prefab in your scene and enabled eye data callback & using the version 2? 

make an empty game object in your scene and add the script on that empty! 

Edited May 12, 2021 by eugie

[luiluks]

Hi eugie,

sorry for the delayed response, but thanks very much for replying! I could figure out my mistake, and you were right, I srewed up the initialization of the sranipal prefab. Now I can successfully gather some data! I'll make shure to include the paper when my projects become more public 😄

[ryanpl89]

Hi guys, 

I've just been following this thread and come up stuck, hopefully someone will be able to offer a bit of support.

I initially tried Eugie script but Unity crashes seems to freeze when I enter game mode.

I then tried Bfalandays script but when I start play mode it then immediately returns to editor mode.

I am using Unity 2020.2.3f1 Does anyone hve any suggestions as what could be causing this?

Thanks

[Justine]

Hello everyone,

I am also encountering the same problem as @ryanpl89.

@eugie and @bfalandays, do you have any suggestions on how to resolve it?

I hope you can help.. Thank you.

[wenbai.xue20]

Hi guys 

I am a beginner on eye-tracking. May I ask some basic question?

How can I read or get some data or information from Sranipal? I use UE4. 

The thing is that I clicked the pack log on SRanipal logo and out put a .zip file, but I cant open or unzip it. Did I do something wrong?

Dose someone may help me on that? Thanks so much.