[Unity] App crashing and laggy tracking on Focus 3 using VideoPlayer

[musk]

Hi, we're developing an app for Vive Focus 3 which plays an interactive video - based on player actions we seek to different timestamps, and ultimately let users replay the video.

However, we ran into following problems, only on the Focus 3 headset:

• after replaying the video several times, the app crashes (usually after ~15 replays)
• occasionally, headset's tracking becomes very laggy, and stays that way even after the app is closed or crashes
  • it seems to happen randomly, and can happen at any point (shortly before app crash, or even during first video playback)
  • seems to happen more often if user takes of the headset, waits for it to automatically 'go to sleep' after a few seconds, and puts it back on
  • in order to fix it, the headset has to be rebooted

 

Here's some of the things we tried/found out so far:

• tested the app on Oculus Quest 2 and Windows/PCVR - none of these issues appear there
• the problems seem to be related to the video playback - calling VideoPlayer methods like Stop() and Prepare() usually results in the app crashing sooner, than when we rewrote the game-logic to only use Pause() method
• no memory usage issues according to Unity Profiler - usage stays the same and there's plenty of free system memory
• doesn't seem to be related to thermals - waiting in the main menu for the headset to cool while the video is paused doesn't change anything
• once head-tracking becomes laggy, it sometimes helps to put the headset into standby mode (by clicking the power button in the rear part of the headset) and waking it up after a couple seconds

 

And here's some more info about the project:

• built with the latest Unity LTS release (2021.3.14) and latest Vive Wave XR Plugin (5.0.3-r.5)
• using Unity's built-in VideoPlayer component to play the video (we also tried AVPro plugin - it fixes the crashing, but it causes visual glitches and is way worse performance in our case)
• video is 4096 x 8192 @ 29.97 frames/second, 180° sterescopic, 8 MBits bitrate, encoded with H.265
• video is rendered into a RenderTexture with color format B10G11R11 (higher color depth options cause terrible performance)

 

I'm also attaching logs from the app.

com.ImmersionVR.Interactive360Experience-logcat.txt

[musk]

I'm attaching some more logs, compressed into a zip because of file size limits:

"log1" - straight logs from one person in QA who experienced the issues, and on restart it began occurring again for them on 2nd playthrough

“log2” - logs from Headset filtered by Unity

“log3” - doesn’t filter by Unity and includes outputs from android lvl logs

Logs.zip

[musk]

I've noticed there's a system update for the Focus today, version 5.0.999.676 - I installed it and re-tested the app, but no changes there.

Can anyone from Vive please comment on the issues I posted above?

[Alex_HTC]

@musk

I'll look at this more in depth later today but wanted to give some quick feedback

I'm not seeing the actual stack trace of the crash even in the full logs

The #1 issue i look for in these cases are the message:
"Decoder will be in frame by frame mode"
which indicates that the low level android decoder will have some performance issues

This showed up in the logs attached, so hopefully we can find a good way to massage the buffers, android library or encoding to get this addressed.

I'm haven't retested unity video player implementation recently on android but there have been a few major overhauls to the android video player that reduce video compatibility, so I'm unsure if they're implemented here. That said, since you're seeking at the edge of 40ms sometimes, try increasing the buffer size and also prevent spamming of the seek to avoid thrashing -- where it tries to re-buffer, which causes a delay, which causes another seek, which causes a delay (repeated) ... until the app is closed due to unresponsiveness

AVPro has options for supporting the newer backends, which may reduce some of these issues. I would try increasing buffer size on these

On the ux side - ensuring that these seeks don't happen too frequently is also likely a

1)The video output of the focus3 is almost twice that of the some competitors, as a result the android library might report errors related to performance (memory bandwidth) in the obtuse way mentioned before "Decoder will be in frame by frame mode".

a)   A brute force way to rule this out is to reduce the eyebuffer size depend on the specifics of the project - XRSettings.eyeTextureResolutionScale,  or in URP 
As mentioned on https://forum.unity.com/threads/windows-mixed-reality-eyetextureresolutionscale-not-working.1097992/ 
b) it is not a silver bullet but for some projects this works and there may be different options needed for this specific project
c) Another is to enable adaptive performance in the settings can help https://hub.vive.com/storage/docs/en-us/WVR_EnableAdaptiveQuality.html
https://hub.vive.com/storage/docs/en-us/UnityXR/UnityXRSettings.html#:~:text=Adaptive Quality Mode,device it is running on.
d) Try using the older methods to change the eye texture buffer

 

3) As another test to rule things out - does encoding in h264 change anything? How about at a lower bitrate?
4) Make sure you're not near the edge of the boundary area, which can happen if set up as sitting only - in the largest projects which push the limits of the hardware, the extra performance overhead of showing the boundary can cause an issue like this
5) It's possible that there's a unity-side memory fragmentation issue, calling GC.Collect every now and again might help. I'm not sure i've seen this with a video project, but the devil is in the details on how the movie buffer is allocated. the most straighforward approach here would be helped by this. I know you mentioned it does not appear to be an issue, but certain types of texture memory does seem to cause(or correlate with) premature out of memory messages in unity.
6) Unlikely, but what audio codecs are being used - there are some really very unusual cases where combinations of audio codecs cause issues, and have been reported to the appropriate parties. No current eta on these, but maybe changing the codec will work to fix the issues


Forgive the messy nature of these notes, glad to better understand, but wanted to provide some context and actionable items quickly and ask questions that help me better understand the issue

Thanks,
Alex

[musk]

Thanks for the (quite lengthy!) response, I'll try these out as soon as possible.

[musk]

 @Alex_HTC Hey, thanks again for looking into it. I tried all Your suggestions, but sadly non of them fixed the issues we're having 😕
Here's the results I got when trying each suggestion:

1) Decrease Render Scale:

• Changing it in the 'Wave XR Settings' did not change the render scale, but using 'XRSettings.eyeTextureResolutionScale' did.
• I managed to set it to as low as 0.1, but it didn't help.
• We're using built-in renderer, but I can try switching to URP as a test later.

 

2) Enable Adaptive Performance Mode:

• We're not using the WaveVR SDK, just the XR plugin, so I don't think this is available to change from code in this case. I can import and test the SDK later.
• There is an 'Adaptive Quality Mode' dropdown in 'Wave XR Settings' that was set to 'Performance Oriented Mode'. Not sure if that works though, since the 'ResolutionScale' slider didn't either.
• I'm attaching a screenshot of the 'Wave XR Settings' we're using, I also highlighted the RenderScale slider and the QualityMode dropdown I mentioned.

 

3) Encode with H264 and/or with lower bitrate:

• Bitrate doesn't seem to have any influence on this. I tried both very high values and very low values, and the issues are present either way.
• As for encoding in H264, I've already tried it before, but re-tested again. It creates some extra problems we could work around / deal with, but it still did not fix the main issues.
  • In the Editor (on Windows), I get an error when trying to play the video: Resolution exceeds the Microsoft Media Foundation H264 decoder limit of 4096 x 2304 on Windows 8 and above.
  • It plays fine on the Focus, with the exception of some playback slowdowns occasionally just after a seek, but other than that it works and looks the same as H265.

 

4) Stay within the boundary:

• Did all the tests within the boundary.

 

5) Call GC.Collect():

• Added a call every time video is finished (to be more specific, after each playthrough / completing the video, we show the user some UI with results and let them retry - so it's a good spot to clear GC).

 

6) Audio codec:

• Audio used is AAC, stereo, 48.0 KHz, Variable bitrate (317 Kbps average, 430 Kbps max), format profile: LC
• As a test, I completely removed the audio stream from the file, as well as set the VideoPlayer in Unity to ignore audio (AudioOutputMode = None)

And here's some clarifications and extra questions I have in regard to Your feedback:

• You mentioned increasing the buffer size for the video player - I don't see a way to do it though. Can you clarify?
• I can take a look at AVPro and see if it's possible to set buffer size there, but I've already tried using AVPro instead of Unity's VideoPlayer. Although it did seem to fix the main issues, it came with a bunch of other problems - bad performance, visual glitches and the colors were slightly off (we're showing a 180 video in the front, as well as a static 180 photo in the back to have a full 360 view, and mismatched colors completely brake the illusion).
• You warned against spamming the seeks - they only happen 2-3 times throughout a single playthrough. Basically, there's one big section (~30s) of the video when player can react (by pressing a button, basically) and then we seek to a corresponding reaction. Then there's one more short (~3s) time range when they can react again, and we may seek to either a completely different part of the video, or one just after that short time range. After that we show some UI with results and let users try again. So in worst-case scenario, two seeks might happen within a second or two of each other.
• The 40ms You mentioned is the time it took for the app to seek to a timestamp (in that particular case, it often takes a bit longer). We're trying to keep these very low, so that the video updates as fast as possible after user's reaction.
• You focused on this specific error: "Decoder will be in frame by frame mode" - from what I can see, it only appeared in one of the log files I attached, and never happened during my tests now. Not sure how relevant it is to these issues.

 

And finally, to add a bit more info about the issues we're having:

• I think the crashes only happen while restarting the video - which means pausing/stopping it, seeking back to beginning, and then trying to play it again. I don't think it ever crashed while seeking somewhere during the video.
• While testing these problems initially, we noticed the crashes happen sooner when we use Stop() method instead of Pause() + seek to beginning. Calling the Prepare() method also seems to have some negative impact on this, but getting rid of it completely still didn't fix the issues.
• In general, the app works fine for the first couple playthroughs, then at some point it usually starts showing some minor video glitches. Finally, in the last couple playthroughs before the crash, it takes noticably longer for the video to start playing when user wants to retry. In the end, the headset freezes for a couple seconds and goes back to the lobby environment.
• The issue with laggy tracking on the other hand is pretty random. Sometimes I can manage to crash the app multiple times and it still doesn't show up, other times it can happen right after first playthrough, on a freshly rebooted headset.

[musk]

Hi, any progress with tackling these issues?

We did some small tweaks/fixes/optimisations on our end, but those only helped in delaying the crashes, and nothing helped with the laggy tracking.

Also, out of these two bugs, laggy tracking is a bit more problematic. When showing a demo to users, we can just restart the app every now and then to prevent it from crashing. But laggy tracking can happen randomly at any point, will make users sick, and it sometimes requires a full headset reboot to fix (as it's also present in the home area/lobby, not just in the app).

Let me know if there's any more info we could provide to help with this.

[Alex_HTC]

@musk Messaged privately, but some way to reproduce this issue to verify. And the version of the system software the f3 is running on (ROM version)

[musk]

Thanks for the quick reply, we'll work on the repro project and reach out to You soon.

[vrunch]

@Alex_HTC 
I'm having the same problem. Help me

All of the optimizations mentioned above have been applied.