Eye Tracking Using Video Player (HTC Vive Pro Eye, Unity, SRanipal)

[Martina_H]

Hi,
I am doing a research project where I would like to gather eye-tracking data while the user is watching a 3D side by side stereoscopic video using a  HTC Vive Pro Eye.
So far, I am using the SRanipal SDK and the HTC Vive Media Decoder Plugin. I tried using the video component of unity but it seems that it can not be focused on (the z-coordinates are very high -> user looks through the video player?). If I add the video player component to a Game Object the video player looses the option to play side by side 3D videos.
Additionally, the position of the gaze direction seems always a bit off when visualizing the gaze point with a dot on the video (when using the decoder plugin).

So my questions are:
1. Am I doing the transformation of the gaze position wrong? (see below - I took parts of the gazeray sample and tobiis gaze visualizer sample)
2. How do I play a 3D side by side video on a Game Object?

Thank you!
Martina

 

 private void Update()
    {
        if (SRanipal_Eye_Framework.Status != SRanipal_Eye_Framework.FrameworkStatus.WORKING &&
            SRanipal_Eye_Framework.Status != SRanipal_Eye_Framework.FrameworkStatus.NOT_SUPPORT) return;

        if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == true && eye_callback_registered == false)
        {
            SRanipal_Eye_v2.WrapperRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye_v2.CallbackBasic)EyeCallback));
            eye_callback_registered = true;
        }
        else if (SRanipal_Eye_Framework.Instance.EnableEyeDataCallback == false && eye_callback_registered == true)
        {
            SRanipal_Eye_v2.WrapperUnRegisterEyeDataCallback(Marshal.GetFunctionPointerForDelegate((SRanipal_Eye_v2.CallbackBasic)EyeCallback));
            eye_callback_registered = false;
        }

        Vector3 GazeOriginCombinedLocal, GazeDirectionCombinedLocal;

        if (eye_callback_registered)
        {
            if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.COMBINE, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal, eyeData)) { }
            else if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.LEFT, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal, eyeData)) { }
            else if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.RIGHT, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal, eyeData)) { }
            else return;
        }
        else
        {
            if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.COMBINE, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal)) { }
            else if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.LEFT, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal)) { }
            else if (SRanipal_Eye_v2.GetGazeRay(GazeIndex.RIGHT, out GazeOriginCombinedLocal, out GazeDirectionCombinedLocal)) { }
            else return;
        }

        Vector3 GazeDirectionCombined = _mainCamera.transform.TransformDirection(GazeDirectionCombinedLocal);
        SetPositionAndScale(GazeOriginCombinedLocal, GazeDirectionCombined);
        _spriteRenderer.enabled = true;
   }

	private void SetPositionAndScale( Vector3 origin, Vector3 direction)
    {
        RaycastHit hit;
        var distance =  _defaultDistance;
        if (Physics.Raycast(origin, direction, out hit))
        {
            distance = hit.distance;
        }

        //var interpolatedGazeDirection = Vector3.Lerp(_lastGazeDirection, direction, Time.unscaledDeltaTime);

        var usedDirection = direction.normalized;

        var tr_pos = _mainCamera.transform.position + usedDirection * distance;
       
        _spriteRenderer.transform.position = tr_pos;//origin + usedDirection * distance;
        
        transform.localScale = Vector3.one * distance * 0.03f;
       

        transform.forward = usedDirection;

        
        _lastGazeDirection = usedDirection;
    }
    

 

[Alex_HTC]

Howdy @Martina_H

1) It looks like your gaze posiiton is correct. The gaze direction is a vector that points from the eye to the point on the screen where the user is looking. When you transform this vector, you are essentially converting it from eye coordinates to world coordinates. This is necessary in order to render the gaze point on the video.

2)To play a 3D side by side video(also referred to as stereoscopic video) on a Game Object, you can use the following steps:

1. Create a new Game Object and add a Video Player component to it.
2. Set the Video Player component's source to the 3D side by side video that you want to play.
3. In the Video Player component's inspector, enable the "Stereoscopic" option and select the "Side by Side" option.
4. Set the Video Player component's z-position to a negative value. This will cause the video to be rendered in front of the user.

An example code snippet for stereo is 
 

        // Create a new Game Object and add a Video Player component to it.
        GameObject videoObject = new GameObject();
        videoObject.AddComponent<VideoPlayer>();
        
        // Set the Video Player component's source to the 3D side by side video that you want to play.
        videoObject.GetComponent<VideoPlayer>().source = VideoSource.Url;
        videoObject.GetComponent<VideoPlayer>().url = "https://www.youtube.com/watch?v=6ZfuNTqbHE8"; //alternatively "file://"+ Application.streamingAssetsPath + "3Dtest.mp4";
        
        // Enable the "Stereoscopic" option - side by side in this example
        videoObject.GetComponent<VideoPlayer>().targetCamera3DLayout = Video3DLayout.SideBySide3D;
        
        // Set the Video Player component's z-position to a negative value.
        videoObject.transform.position = new Vector3(0, 0, -1);

Note that the path to the video will be relative the the build - so if you want to build and run the app and always have it include the video, you could put the video inside the streaming assets path in your project and then use the Application.streamingAssetsPath to reference the folder name as referenced ehere https://docs.unity3d.com/ScriptReference/Application-streamingAssetsPath.html . Interestingly enough, that an example showing how to use new unity video player component as well at that link.

u could use the standard vive media decoder, split the material down the middle yourself and show one half on one eye and another on another eye using layer masks and make a vr rig with two eyes, one for the left and one for the right, with the culling mask excluding the other eye layer

Thanks,
Alex

[Martina_H]

@Alex_HTC Thank you for your quick reply! I tried the script and steps you provided to play a stereoscopic video on a Game Object but it does not seem to work. I am seeing only parts of the Game Object (and the Gaze Point) behind the Video playing or not at all. Also I do not see an option to adapt the Video Player Components size. Is there maybe a way to work with render textures? I have already tried this one (https://github.com/Unity-Technologies/SkyboxPanoramicShader) but I can not attach it to a GameObject and the quality of the video is very poor.

The HTC ViveMediaDecoder Plugin ( https://developer.vive.com/resources/vive-sense/tools/vive-media-decoder/) seems to have ability to split down the material on its own and also requires one to use layer masks etc. (as specified in their readme). I followed their instructions but sadly the quality is not good and the video seems not to be 3D. Do you know why this is the case? 

[llz]

hi, I have the same research goal as you. But first of all, I have a question. Is there any open source SDK that allows me to obtain the trajectory of head movement and eye gaze? My device is HTC VIVE Pro Eye. 

[Martina_H]

@llzHi! I am using the Sranipal sdk https://developer.vive.com/resources/downloads/

@Alex_HTC I have another question regarding the transformation from local to world coordinates: Because I am using the HTC ViveMediaDecoder I have two cameras to create the 3D illusion and no main camera. For the transformation of the eye_data does it matter if I use the left_eye_camera or the right_eye_camera? Can I use both? If yes how?

Thanks!

Martina

[maryam]

Dear Support Team,

I am currently using the HTC Vive Pro Eye headset and would like to view 360-degree and interactive virtual tours created with 3DVista. Additionally, I would like to take advantage of the Eye Tracking feature available in the Vive Pro Eye for an enhanced viewing experience.

Could you please guide me on the best way to export content from 3DVista and view it on the HTC Vive Pro Eye, utilizing the eye-tracking feature?

My questions are as follows:

1. What type of export or file format from 3DVista should I use to ensure compatibility with the HTC Vive Pro Eye?
2. Does the eye-tracking feature in HTC Vive Pro Eye support 360-degree and interactive tours? If so, how can I enable and use this feature effectively?
3. Are there any specific software or applications I need to use to view interactive tours or 360-degree videos with eye-tracking functionality?
4. How can I view interactive virtual tours in VR? Do I need to use a specific browser or upload files in a particular format?
5. Are there any special settings in 3DVista or the HTC Vive Pro Eye required for utilizing eye-tracking and displaying the content correctly?
6. Is it possible to directly play 360-degree videos on the HTC Vive Pro Eye while utilizing eye-tracking functionality? If so, how can I set this up?

I would greatly appreciate any guidance you can provide to help me get the most out of the HTC Vive Pro Eye with 3DVista content, especially regarding the eye-tracking feature.

Thank you for your support!

[VIVE_chengnay]

On 1/25/2025 at 12:41 AM, maryam said:

Could you please guide me on the best way to export content from 3DVista and view it on the HTC Vive Pro Eye, utilizing the eye-tracking feature?

Hi @maryam,

May I know 3DVista is a free or paid software?

[maryam]

22 hours ago, VIVE_chengnay said:

Hi @maryam,

May I know 3DVista is a free or paid software?

3DVista is a paid software used for creating virtual tours, 360-degree walkthroughs, and interactive multimedia content. It offers various versions, including 3DVista Virtual Tour Pro, which requires purchasing a license. However, a free trial version is available with certain limitations, such as watermarks on projects or restricted access to features.

The software supports multiple output formats:

1. Web Output: Publish tours online for use on browsers across devices, with hosting options on the 3DVista cloud or your own server.
2. Offline Output: Export standalone applications for Windows, Mac, iOS, and Android, ideal for offline environments like trade shows or kiosks.
3. VR Mode: Create tours for virtual reality devices, including Oculus, HTC Vive, and Pico.
4. 360 Video: Export as 360-degree videos for platforms like YouTube or Vimeo.
5. Custom Embedding: Generate HTML code to embed tours directly into websites or applications.
6. High-Resolution Panoramas: Export panoramic images for printing or presentations.

These output options make 3DVista highly versatile for industries like real estate, tourism, education, and marketing. For more information on pricing and features, visit the official 3DVista website.

[VIVE_chengnay]

47 minutes ago, maryam said:

3DVista is a paid software used for creating virtual tours, 360-degree walkthroughs, and interactive multimedia content

Hi @maryam,

If it is paid software, it is hard for me to try and answer to your questions. Could you ask 3DVista for support?

[niva]

 Inquiry Regarding Eye-Tracking Data Recording in HTC Vive Pro Eye

Dear HTC Support Team,

We have acquired an HTC Vive Pro Eye headset for a research project aimed at analyzing eye-tracking data during user interactions with:

1. 360-degree video content
2. A virtual reality tour developed in an HTML-based environment

✅ Current Status:

• We have successfully completed the eye-tracking calibration and verified its functionality.
• We now seek a method to capture and store eye-tracking data while users engage with these two types of content.

We would appreciate your guidance on the appropriate approach to achieve this.

Thank you for your assistance.