eye tracking Getting Screen Position of the Gaze in Unity

[AdamStreck]

Update - Unity has reproduced the issue on their end and flagged a bug: https://issuetracker.unity3d.com/issues/xr-sdk-cameraworldtoscreenpoint-returns-offset-coordinates

[Corvus]

@AdamStreck Thanks for sharing the update!

[neomimiz]

 

Hi,

I have recently started working with unity and Sranipal.  I have tried to follow the steps described in the recent posts here but still have a few questions related to the conversion between gaze direction screen and world coordinates...

1. In what way gaze direction is related to the screen coordinates in unity? is this the correct conversion to pixels?

  Left_E_gaze_direction_normalized = verboseData.left.gaze_direction_normalized;

x_pix_resolution=1440;  y_pix_resolution=1600;

 left_eye_x_pixels = Left_E_gaze_direction_normalized.x / x_pix_resolution;
  left_eye_z_pixels = Left_E_gaze_direction_normalized.z / y_pix_resolution;

2. Conversion to world coordinates:

Left_eye_position_world_coordinates= Camera.ScreenToWorldPoint( left_eye_x_pixels, left_eye_z_pixels, left_eye_y_pixels);

3. In what way gaze origin is related to these transformations? /or if it isn't why is it provided? 

I would greatly appreciate your help:)

Neomi

 

 

 

 

[scyudits]

I am also interested in the answers to the above post by @neomimiz

[Playnary]

 

On 4/5/2021 at 8:14 AM, scyudits said:

I am also interested in the answers to the above post by @neomimiz

Me too!

[michael_wang]

Hi, @neomimiz @scyudits @Playnary

Validity of the data: 

 

 

If I understand correctly, the following shows how it works, though I am not sure if this is correct.

Enumerator	Binary digit if valid	Decimal digit if valid
SINGLE_EYE_DATA_GAZE_ORIGIN_VALIDITY	00001	1
SINGLE_EYE_DATA_GAZE_DIRECTION_VALIDITY	00010	2
SINGLE_EYE_DATA_PUPIL_DIAMETER_VALIDITY	00100	4
SINGLE_EYE_DATA_EYE_OPENNESS_VALIDITY	01000	8
SINGLE_EYE_DATA_PUPIL_POSITION_IN_SENSOR_AREA_VALIDITY	10000	16

Calculation of visual angle:

Are you asking how to convert a GAZE vector into a rotated vector?
we have sample code for this In SDK unity and unreal sample plugin  
AvartarSRanipal_v2.cs line:189

Gaze origin and pupil position: 
These are the values generated by Tobi, I think they should be used to calculate the gaze vector

[Drifter]

On 11/6/2020 at 1:03 AM, AdamStreck said:

I did not figure it out. Would you please post details @Av ?

I know where the issue lies, though, it's a difference between the view provided by "Left Eye" vs the full image rendered to the headset.

I have made a simple example, where the camera coordinates of an object are printed directly on screen. The project is attached (EyeTrackingIssue.zip), but the full code is the following:
 

using UnityEngine;
using UnityEngine.UI;
using UnityEngine.XR;

public class DisplaySSCoordinates : MonoBehaviour
{
    public Transform debugSphere;
    public Text displayText;
    public Camera vrCamera;
    
    private void Update()
    {
        var coords = vrCamera.WorldToScreenPoint(debugSphere.position, Camera.MonoOrStereoscopicEye.Left);
        var eyeTextureSize = new Vector2(XRSettings.eyeTextureWidth, XRSettings.eyeTextureHeight);
        var units = new Vector2(coords.x / eyeTextureSize.x, coords.y / eyeTextureSize.y);
        displayText.text = $"({units.x:F2},{units.y:F2})";
    }
}

You can see that in the bottom-left and top-right corners, the coordinates are not what I would expect. For the center it is correct:

However when I switch from the "LeftEye" to "OcclusionMesh", the coordinates are correct:

This is obviously not SRanipal issue only, probably will also happen for other EyeTracking systems, however I think this is the prescribed way of getting the screen-space position (it's a while I'm not sure, I'm happy for a different workflow).

I noticed that the error changes with the aspect ratio, which makes sense as the screen-crop of the "LeftEye" changes too.

I'd need to obtain the information how the "EyeTexture" is cropped into the game view to calculate the position correctly. Any recommendations?



 

 

EyeTrackingIssue.zip 23.88 kB · 26 downloads

Hi, I had a similar problem with my recent development. Where is the "OcclusionMesh" you mentioned turned on or off? I hope to be able to get at least the exact screen coordinates of the eye in this way. Looking forward to your reply.