Hand Tracking's need to support Unreal Engine

[duelgame]

I've tested Vive Hand Tracking in Unreal Engine 4.,24

CPU : I7-7700k
GPU : GTX 1660
RAM : 32Gb
Vive Cosmos (https://www.vive.com/us/product/vive-cosmos/overview/)
ViveHandTracking 0.9.2

 

I saw this post (https://forum.vive.com/topic/8459-vive-hand-tracking-camera-resolution/#) and was wondering if only Unreal had issues with performance. So I tested the Unity samples using RenderDoc. Here are the results below:

 

Unity sample used two textures of 1584x1872.

It rendered the Left Eye and Right Eye. separately.

Basically Unreal Engine uses Instanced Stereo Rendering.

Even if I turn off the Instanced Stereo Rendering, the Resolution is not changed.

I think VIVE hand tracking uses image processing so it's performance is greatly affected by the resolution it processes.

The performance is fine when I test Unreal Sample with vr.PixelDensity 0.5

 

Unity has a similar option called single pass stereo rendering.

https://docs.unity3d.com/Manual/SinglePassStereoRendering.html

So I think VIVE needs to support Instanced Stereo Rendering and Single Pass Stereo Rendering to improve the overall FPS.

I need to know your thoughts on this and whether you have plans to update the SDK to improve performance. 

 

 

 

 

 

 

[zzy]

Hi @duelgame

I cannot see most of the pictures you attached, but here is my answer based on text in your question.

To make it clear, here is hand tracking process: Camera frame -> Hand Tracking Process (GPU) -> output result in engine plugin. This has nothing to do with the render texture size, or vr.PixelDensity.

However, VR rendering also uses GPU. Hand Tracking and render shares the GPU resource, so if you reduce workload for either side, you can observe performance improvement. This is case when you reduce rendering workload (i.e. by reducing vr.PixelDensity or using Instanced Stereo Rendering). You can observe the same thing by using 3d point mode instead of skeleton mode in Vive Hand Tracking.

I'm not expert for Unreal, but I think you need to check SteamVR plugin for Instanced Stereo Rendering support in Unreal.

[duelgame]

@zzy

I attached a PDF file for the images above in case you don't see them.

There seems to be no performance increase when I use 3d point mode .

Only rendering a yellow plane, two blue hands that are using basic shaders and few triangles shouldn't be consuming much GPU resources... 

How can I make a simple vr game with this if the performance is so low?

If this is an Unreal issue, there should be someone that knows Unreal in the VIVE team that is able to handle these type issues. 

 

01.pdf

[zzy]

Hi @duelgame

I can see all the images now, I think both Unreal and Hand Tracking need to optimize GPU usage:

1. Next release version of Vive Hand Tracking will improve GPU usage (target end of this month).
2. From a quick google search, I found Instanced Stereo Rendering support is already in VR: https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/VRPerformance/index.html. Is there any reason that you cannot enable this feature?

We will keep improving GPU usage in future versions, but please understand currently improving accuracy has higher priority.

 

[zzy]

Hi @duelgame

v0.9.3 is now released, you can download it from here. It improves GPU usage in hand tracking part. Please have a try.