cte
-
Posts
21 -
Joined
-
Last visited
Content Type
Profiles
Forums
Gallery
Blogs
Events
Store
Downloads
Posts posted by cte
-
-
Hi,
Recently I've been needing to find the local address (think IP or similar) for the eye tracker in the HMD to extract or write data gathered there direct to an application that relies on R for data processing.
Is there a host address for the eye tracker presently that I can call in the SRanipal SDK?
-
UPDATE:
🎉🎉Found the fix and now have functional dilation extraction going. 🎉🎉
-----------------------------------------------------------
SRanipalEye_FunctionLibrary.cpp
bool USRanipalEye_FunctionLibrary::GetPupilDiameter(EyeIndex eye, float& diameter_mm) { return SRanipalEye_Core::Instance()->GetPupilDiameter(eye, diameter_mm); }SRanipalEye_Core.cpp
bool SRanipalEye_Core::GetPupilDiameter(EyeIndex eye, float &diameter_mm) { bool valid = false; if (SRanipalEye_Framework::Instance()->GetStatus() == SRanipalEye_Framework::FrameworkStatus::NOT_SUPPORT) { valid = true; diameter_mm = 0.0f; } else { UpdateData(); Eye::SingleEyeData eyeData; switch (SRanipalEye_Framework::Instance()->GetEyeVersion()) { case SupportedEyeVersion::version1: eyeData = eye == EyeIndex::LEFT ? EyeData_.verbose_data.left : EyeData_.verbose_data.right; break; case SupportedEyeVersion::version2: eyeData = eye == EyeIndex::LEFT ? EyeData_v2.verbose_data.left : EyeData_v2.verbose_data.right; break; } valid = eyeData.GetValidity(Eye::SingleEyeDataValidity::SINGLE_EYE_DATA_PUPIL_DIAMETER_VALIDITY); if (valid) { diameter_mm = eyeData.pupil_diameter_mm; } else diameter_mm = 0.0f; } return valid; }SRanipal_Core.h
bool GetPupilDiameter(EyeIndex eye, float& diameter_mm);SRanipalEye_FunctionLibrary.h
UFUNCTION(BlueprintCallable, Category = "SRanipal|Eye") static bool GetPupilDiameter(EyeIndex eye, float& diameter_mm);-
2
-
-
@Franka & @MariosBikos_HTC thanks for the help.
For reference, I'm working off of the latest version of the SRanipal Framework so had to alter some of the code.
Now I'm having a little trouble understanding what I need to feed into the blueprints I'm calling and assume I have put something wrong in here as I want diameter MM as a string output rather than an input so I can write it to a csv.
Here is the function in blueprints:
Question: What do I need to alter to get the diameter Mm as a string output variable?> I'm going to look into this but would love a response here if someone knows how this would.
- I've looked into this a little bit and it seems that the absence of the framework is what is really creating issues here.
Question: Based on the updates to SRanipal SDK what do I call instead of the framework for getting the data?
----------------------------------------------------------------------------------------------------------------------------------
👇👇👇 Changed Code 👇👇👇👇
➡️ SRanipalEye_Core.cpp:
Added:
bool SRanipalEye_Core::GetPupilDiameter(EyeIndex eye, float& diameter_mm) { bool valid = false; if (SRanipalEye_Framework::Instance()->GetStatus() == SRanipalEye_Framework::FrameworkStatus::NOT_SUPPORT) { valid = true; diameter_mm = 0.0f; } else { UpdateData(); Eye::SingleEyeData EyeData = eye == EyeIndex::LEFT ? EyeData_.verbose_data.left : EyeData_.verbose_data.right; valid = EyeData.GetValidity(Eye::SingleEyeDataValidity::SINGLE_EYE_DATA_PUPIL_DIAMETER_VALIDITY); if (valid) { diameter_mm = EyeData.pupil_diameter_mm; } else { diameter_mm = 0.0f; } } return valid; }➡️ SRanipalEye_FunctionLibrary.h
Added:
UFUNCTION(BlueprintCallable, Category = "SRanipal|Eye") static bool GetPupilDiameter(EyeIndex eye, UPARAM(ref) float& diameter_mm);➡️ SRanipalEye_FunctionLibrary.cpp
Added:
bool USRanipalEye_FunctionLibrary::GetPupilDiameter(EyeIndex eye, UPARAM(ref) float& diameter_mm) { return SRanipalEye_Core::Instance()->GetPupilDiameter(eye, diameter_mm); } -
@Franka & @MariosBikos_HTC thanks for the help.
For reference, I'm working off of the latest version of the SRanipal Framework so had to alter some of the code.
Now I'm having a little trouble understanding what I need to feed into the blueprints I'm calling and assume I have put something wrong in here as I want diameter MM as a string output rather than an input so I can write it to a csv.
Here is the function in blueprints:
Question: What do I need to alter to get the diameter Mm as a string output variable?
> I'm going to look into this but would love a response here if someone knows how this would.
----------------------------------------------------------------------------------------------------------------------------------
👇👇👇 Changed Code 👇👇👇👇
➡️ SRanipalEye_Core.cpp:
Added:
bool SRanipalEye_Core::GetPupilDiameter(EyeIndex eye, float& diameter_mm) { bool valid = false; if (SRanipalEye_Framework::Instance()->GetStatus() == SRanipalEye_Framework::FrameworkStatus::NOT_SUPPORT) { valid = true; diameter_mm = 0.0f; } else { UpdateData(); Eye::SingleEyeData EyeData = eye == EyeIndex::LEFT ? EyeData_.verbose_data.left : EyeData_.verbose_data.right; valid = EyeData.GetValidity(Eye::SingleEyeDataValidity::SINGLE_EYE_DATA_PUPIL_DIAMETER_VALIDITY); if (valid) { diameter_mm = EyeData.pupil_diameter_mm; } else { diameter_mm = 0.0f; } } return valid; }➡️ SRanipalEye_FunctionLibrary.h
Added:
UFUNCTION(BlueprintCallable, Category = "SRanipal|Eye") static bool GetPupilDiameter(EyeIndex eye, UPARAM(ref) float& diameter_mm);➡️ SRanipalEye_FunctionLibrary.cpp
Added:
bool USRanipalEye_FunctionLibrary::GetPupilDiameter(EyeIndex eye, UPARAM(ref) float& diameter_mm) { return SRanipalEye_Core::Instance()->GetPupilDiameter(eye, diameter_mm); } -
47 minutes ago, cte said:
I was able to get this to work now in UE4 and what it took was learning a bit about actors and blueprints.
To get gaze you can create a new actor and place it in the scene, then create a blueprint on that actor.
For getting gaze data I produced the following blueprint:
A key feature of this blueprint was the File IO Save String Arrray which I got from a function library plugin called "Victory Plugin" which you can get here: https://www.mediafire.com/file/pgrpsf9my7tv1ae/VictoryPlugin25.zip/file
Though this doesn't solve my pupil dilation problem, I can probably get that data via a functionb posted in a different forum:
I'll be trying that code next and creating a new actor and blueprint for that particular function.
The data I'm getting now looks like this:
From the CSV file. I'd like to get this in a different format where things are columns rather than rows but I can probably work wiht this data and manipulate it to that effect in r or something I know better than C++ functions.
Anyway, that is the update on the extracting data from SRanipal for eye direction, fixation, and confidence value with a unix timestamp.
Would love to get a few thoughts on how to improve this blueprinting setup.
-CTE
UPDATED Blueprint :: Found the Join String Array which Makes a significantly easier to work with CSV file.
New Blueprint Structure:
Leads to new output in csv file:
-
1
-
-
I was able to get this to work now in UE4 and what it took was learning a bit about actors and blueprints.
To get gaze you can create a new actor and place it in the scene, then create a blueprint on that actor.
For getting gaze data I produced the following blueprint:
A key feature of this blueprint was the File IO Save String Arrray which I got from a function library plugin called "Victory Plugin" which you can get here: https://www.mediafire.com/file/pgrpsf9my7tv1ae/VictoryPlugin25.zip/file
Though this doesn't solve my pupil dilation problem, I can probably get that data via a functionb posted in a different forum:
I'll be trying that code next and creating a new actor and blueprint for that particular function.
The data I'm getting now looks like this:
From the CSV file. I'd like to get this in a different format where things are columns rather than rows but I can probably work wiht this data and manipulate it to that effect in r or something I know better than C++ functions.
Anyway, that is the update on the extracting data from SRanipal for eye direction, fixation, and confidence value with a unix timestamp.
Would love to get a few thoughts on how to improve this blueprinting setup.
-CTE
-
1
-
-
:: UPDATE :: 10-07-20 ::
Application:
- Academic Research
Goals:
- Install SDK ----------------------------- [ X ]
- Get Eye Gaze ------------------------- [ X ]
- Get Fixation --------------------------- [ ]
- Get Pupil Dilation ------------------- [ ]
- Run Subjects & Get Tenure ----- [ ]
Question:
- How do I reference the SDK's framework / API to extract close to real-time eye tracking data that prints either in a data.frame or CSV file.
---------------------------------------------------------------------------------------------------
I've given up (more or less) on getting this to work in UE4 so tried out the sample code for _C in the SDK and had a bit more success getting data.
I can now get gaze data from the sample project in some fashion but have to figure out now how to get dilation data which might mean I do a similar kind of thing to what @MariosBikos_HTC did here:
I'm not sure how to wrap my head around this function library code but am assuming this is a type of blueprint class reference in unreal that I can recycle for calculating the same information and providing it to me in a string format in C.
Not sure about this though since I'm not much a C programming language user but I will try to create a new header and call the function in the .cpp file such that it will show up as a keyed option "7" to start the string and "8" to stop it. Doing so, I think I can just mirror the existing gaze data keys but again, I'm not really sure I know exactly what I'm doing so appreciate any guidance in this process.
-
2 minutes ago, nbhatia said:
Hi, have you checked the sdk? There are dome examples already.
Trying to work trough it now. I see the different levels in Unreal but don't know where to start with the data extraction. I examined the dartboard actor but am unsure how to use what I'm looking at here (DartBoard.h)
---------------------------------
Do I need to modify this code so as to utilize it as a kind of template?
- If so, where do I modify and what structure do I use?
// ========= Copyright 2019, HTC Corporation. All rights reserved. =========== #pragma once #include "SRanipal_FunctionLibrary_Eye.h" #include "SRanipal_Eye.h" #include "Engine/Classes/Camera/PlayerCameraManager.h" #include "Runtime/Engine/Classes/Materials/Material.h" #include "CoreMinimal.h" #include "GameFramework/Actor.h" #include "DartBoard.generated.h" UCLASS() class SRANIPAL_API ADartBoard : public AActor { GENERATED_BODY() public: // Sets default values for this actor's properties ADartBoard(); UPROPERTY(EditAnywhere) FVector Position; // The dartboard's position UPROPERTY(EditAnywhere) UStaticMeshComponent* BoardMesh; UPROPERTY(EditAnywhere) UMaterial* ParentMaterial; UPROPERTY() APlayerCameraManager* PlayerCameraRef; private: FVector FocusPosition; UMaterialInstanceDynamic* DartBoardMaterial; protected: // Called when the game starts or when spawned virtual void BeginPlay() override; public: // Called every frames virtual void Tick(float DeltaTime) override; private: // Focus relative parameter FFocusInfo FocusInfo; FVector GazeOrigin, GazeDirection; float RayLength = 1000.0f; float RayRadius = 1.0f; // Move dart board FVector InitialPosition; bool FirstUpdate = true; bool AlreadyMoveDart = false; void MoveDartBoard(); float SignedAngle(FVector v1, FVector v2, FVector v_forward); };Sorry if this is a basic question but I'm brand new to C++ 👶
-
@Corvus would you perhaps be willing to help out with this low-level explanation of how to get at the SDK's data? I'm new to C++ and compiler languages in general so don't know how to call the API (if that is even how this works).
-
@nbhatia how are you going about getting the data from the SDK? I'm new to this and trying to figure out how to pull that data into a writable export file of some kind (ideally a CSV).
-
Application:
- Academic Research
Goals:
- Install SDK ----------------------------- [ X ]
- Get Eye Gaze ------------------------- [ ]
- Get Fixation --------------------------- [ ]
- Get Pupil Dilation ------------------- [ ]
- Run Subjects & Get Tenure ----- [ ]
Question:
- How do I reference the SDK's framework / API to extract close to real-time eye tracking data that prints either in a data.frame or CSV file.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi there, I've been able to get the VR and eye tracking set up and working per a previous thread with UE 4.25.3 and have a demo with VR functional.
The only thing is I'm not sure where to go from here to get the SDK to write/print data to a csv or similar file.
Presently I don't need any eye tracking interactions within the VR environment so the dartboards and mannequin head are useful in that they show this data exists but I need to get that data out of whatever loop it is in and write it to a data file for processing in statistical programs and the like.
@MariosBikos_HTC , you've been a great help so far. Let me know if you or another HTC fellow are the right ones to ask about this function.
Once I get something functional I'll definitely be sharing it for future folks in my position.
-
@Franka I'm trying to get some pupil dilation data as well.
I wanted to run this fix but wasn't sure what you meant by header file.
I'm also thinking of extracting this data and writing to a file of some kind for processing in R or Python.
Sounds like the detection isn't working great though so I'm going to try my hand at figuring it out and will let you know if I can get something.
-CTE
-
Working off of the new errors I reported above, I was able to get everything working and the dartboards and eye tracking functional.
For the first chunk of similar errors:
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample.cpp 107
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample.cpp 108
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample_v2.cpp 107
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample_v2.cpp 108
I applied the earlier fix provided in this thread where you:Replace
RelativeLocationWith
GetRelativeLocation()
Where they appeared in the code.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Upon fixing this, I was faced with "Fatal Error C1083"
This was linked to the XXX_XXX_Lip.h series of files found both in "Source/SRanipal/Public/Lip" and "Source/SRanipal/Private/Lip" folders.
This error was caused by the compiler not being able to locate tehse files. To fix this I simply added the exact directory location to the points where the errors were flagging. So, for example if I saw:
#Include "SRanipal_AvatarLipSample.h"I would replace this with
#include "SRanipal/Public/Lip/SRanipal_AvatarLipSample.h"
This fix took a few iterations of building and fixign the error until I built and there were no errors.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Here is the working source folder that may contain all the fixes needed to make things work.
This can replace the existing source folder in: \Plugins
After all that I was able to ge the dartboards and eye balls in the head working well enough.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Next I'm going to look at extracting pupil dialation which I think is maybe convered in:
-
1
-
-
Applied fix detailed by @MariosBikos_HTC above and additional errors were triggered.
=======================================================
Fix Applied to: ~\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\SRanipal.Build.cs
Modified SRanipal.Build.cs Attached.
---------------------------------------------------------------------------------------------------
New Errors
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample.cpp 107
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample.cpp 108
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample_v2.cpp 107
QuoteError C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent' MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\SRanipal_AvatarEyeSample_v2.cpp 108
QuoteError C1076 compiler limit: internal heap limit reached MyProject Z:\Unreal Projects\MyProject\Intermediate\ProjectFiles\c1xx 1
QuoteError C3859 Failed to create virtual memory for PCH MyProject Z:\Unreal Projects\MyProject\Intermediate\ProjectFiles\c1xx 1
QuoteWarning Library 'SRanipal.lib' was not resolvable to a file when used in Module 'SRanipal', assuming it is a filename and will search library paths for it. This is slow and dependency checking will not work for it. Please update reference to be fully qualified alternatively use PublicSystemLibraryPaths if you do intended to use this slow path to suppress this warning. MyProject Z:\Unreal Projects\MyProject\Intermediate\ProjectFiles\UnrealBuildTool 1
QuoteError MSB3073 The command "C:\Windows\System32\chcp.com 65001 >NUL
"Z:\Epic Games\UE_4.25\Engine\Build\BatchFiles\Rebuild.bat" MyProjectEditor Win64 Development -Project="Z:\Unreal Projects\MyProject\MyProject.uproject" -WaitMutex -FromMsBuild" exited with code -1. MyProject Z:\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.MakeFile.Targets 54 -
Downloaded the latest version of the SDK on 09/28/20 and I am still seeing a similar errors I was seeing before this release.
Download URL: https://hub.vive.com/en-US/download
----------------------------------------------------------------------------------------------------------------
ERRORS and Screen-caps:
QuoteSeverity Code Description Project File Line Suppression State
Error C1083 Cannot open include file: 'DartBoard.h': No such file or directory MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\DartBoard.cpp 3
-------------------------------------------------------------------------------
QuoteSeverity Code Description Project File Line Suppression State
Error MSB3073 The command "C:\Windows\System32\chcp.com 65001 >NUL
"Z:\Epic Games\UE_4.25\Engine\Build\BatchFiles\Build.bat" MyProjectEditor Win64 Development -Project="Z:\Unreal Projects\MyProject\MyProject.uproject" -WaitMutex -FromMsBuild" exited with code 6. MyProject Z:\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.MakeFile.Targets 46
-
NICE! I'll hang out till the 24th then since it isn't much more of a wait.
Just found some time today between playing with a Shimmer GSR to give this another go.
I'll update this forum after the release and see if it is working for me next time I have a chance to dig into the eye tracking SDK.
Thanks for the help so far 🙂
-
@MariosBikos_HTC has the update for this released?
I'm trying a re-downloaded version of the SDK and am still getting some errors that are different than those you've helped me with above.
Namely:
QuoteSeverity Code Description Project File Line Suppression State
Error C1083 Cannot open include file: 'DartBoard.h': No such file or directory MyProject Z:\Unreal Projects\MyProject\Plugins\SRanipal\Source\SRanipal\Private\Eye\DartBoard.cpp 3
AND
QuoteSeverity Code Description Project File Line Suppression State
Error MSB3073 The command "C:\Windows\System32\chcp.com 65001 >NUL
"Z:\Epic Games\UE_4.25\Engine\Build\BatchFiles\Build.bat" MyProjectEditor Win64 Development -Project="Z:\Unreal Projects\MyProject\MyProject.uproject" -WaitMutex -FromMsBuild" exited with code 6. MyProject Z:\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.MakeFile.Targets 46
--------------------------------------------------
Please let me know if I'm doing something wrong on my end.
The process has been as follows:
1. Start new blank C++ project in UE4.25.3
2. Close project and paste in extracted plugin from: https://hub.vive.com/en-US/download found in "SRanipal_version\03_Unreal\Vive-SRanipal-Unreal-Plugin.zip"
3. Start project
-- Error State --
QuoteMyProject Could not be compiled. Try rebuilding them from the source manually.
4. Opened Visual Studio file and go to build.
-- Error States --
<See above>
-------------------------------------------------
Best,
Connor Esterwood
-
Awesome, I'll give that a shot tomorrow some time and see if I can't get things running.
Thanks for the prompt reply and super helpful visuals 🙌🙌🙌
-CTE
-
1
-
-
@MariosBikos_HTC Maybe you could be of help here since I'm following the revisions you did to the plugin but it still isn't working. Thanks btw if you find the time.
-
Problem
- I'm having some difficulty getting the plugin for unreal engine installed for the Vive Pro Eye Headset.
- Following the directions in the documentation leads to compile errors when I go to launch the project.
- In short, I can't get the plugin to compile or the project to load correctly.
Done:
- [X] - Installed and running steam VR
- [X] - Installed and running SR_Runtime
- [X] - Calibrated Eye Tracker in Steam VR
Error Trigger
- Pasted unzipped unreal plugin into C++ blank UE4 project.
- Returned message from unreal:
Quote
MyProject Could not be compiled. Try rebuilding them from the source manually.
-
Result of manual compilation via project.sln
- Menu -> Build -> Build Solution
Quote
Error MSB3073 The command "C:\Windows\System32\chcp.com 65001 >NULC2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent'
C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent'
C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent'
C2248 'USceneComponent::RelativeLocation': cannot access private member declared in class 'USceneComponent'
--------------------------------------------------------
I'm also trying to follow the directions in part 3 of :
------------------------------------------------------------
For Context:
- I'm a researcher at the University of Michigan and have a series of studies I would like to gather eye tracking data from (especially blink rate and pupil dilatation). We are using a fully developed VR environment in UE4 to give several treatments to participants.
Local Address for API or Tracker
in VIVE Eye and Facial Tracking SDK
Posted
Thanks for getting back!
I was interested in getting a more direct link to the eye tracker in the vive than is currently offered in the UE4 plugin.
The idea is to build non-engine specific data pipelines so I can extract dilation and gaze (using ROS in my case).