Scene Understanding Tutorial and Sample not working?

[maxxsby]

Hi, I wanted to ask about why does the Scene Understanding Example not working out for me. I am using Unity 2021.3.2f1 and have also added the OpenXR Vive Plugin, but still cannot play the demo sample. Then I tried to not use the sample and just make a custom one like the tutorial, but the script kept on disabling itself because it cannot found any XRMeshSubsystem using SubsystemManager.GetInstances(). Is there anything that I'm missing??

Oh yeah, this is also the setting for OpenXR Plugin Management

Here's my script for MeshingTeapotFeature

using System;
using System.Collections;
using System.Collections.Generic;
using System.Runtime.InteropServices;

using UnityEditor;
using UnityEditor.XR.OpenXR.Features;

using UnityEngine;
using UnityEngine.XR;
using VIVE.SceneUnderstanding;

namespace UnityEngine.XR.OpenXR.Samples.MeshingFeature
{
    #if UNITY_EDITOR
    [
        OpenXRFeature
        (
            UiName = "Meshing Subsystem",
            BuildTargetGroups = new[] { BuildTargetGroup.Standalone, BuildTargetGroup.WSA, BuildTargetGroup.Android },
            Company = "HTC",
            Desc = "Example extension showing how to supply a mesh from native code with OpenXR SceneUnderstanding functions.",
            DocumentationLink = "https://developer.vive.com/resources/openxr/openxr-pcvr/tutorials/unity/interact-real-world-openxr-scene-understanding/",
            OpenxrExtensionStrings = "",
            Version = "0.0.1",
            FeatureId = featureId
        )
    ]
    #endif

    public class MeshingTeapotFeature : SceneUnderstanding_OpenXR_API
    {
        public new const string featureId = "com.unity.openxr.feature.example.meshing";
        private static List<XRMeshSubsystemDescriptor> s_MeshDescriptors = new();

        protected override void OnSubsystemCreate()
        {
            CreateSubsystem<XRMeshSubsystemDescriptor, XRMeshSubsystem>(s_MeshDescriptors, "Sample Meshing");
        }

        /// <inheritdoc />
        protected override void OnSubsystemStart()
        {
            StartSubsystem<XRMeshSubsystem>();
        }

        /// <inheritdoc />
        protected override void OnSubsystemStop()
        {
            StopSubsystem<XRMeshSubsystem>();
        }

        /// <inheritdoc />
        protected override void OnSubsystemDestroy()
        {
            DestroySubsystem<XRMeshSubsystem>();
        }
        protected override void OnSessionCreate(ulong xrSession)
        {
            m_XrSession = xrSession;

            NativeApi.SetOpenXRVariables(m_XrInstance, m_XrSession,
                Marshal.GetFunctionPointerForDelegate(m_XrEnumerateReferenceSpaces),
                Marshal.GetFunctionPointerForDelegate(m_XrCreateReferenceSpace),
                Marshal.GetFunctionPointerForDelegate(m_XrDestroySpace),
                Marshal.GetFunctionPointerForDelegate(m_XrEnumerateSceneComputeFeaturesMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrCreateSceneObserverMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrDestroySceneObserverMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrCreateSceneMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrDestroySceneMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrComputeNewSceneMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrGetSceneComputeStateMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrGetSceneComponentsMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrLocateSceneComponentsMSFT),
                Marshal.GetFunctionPointerForDelegate(m_XrGetSceneMeshBuffersMSFT));
            systemProperties.type = XrStructureType.XR_TYPE_SYSTEM_PROPERTIES;
            XrSystemPassThroughPropertiesHTC SystemPassThroughPropertiesHTC;
            SystemPassThroughPropertiesHTC.type = XrStructureType.XR_TYPE_SYSTEM_PASS_THROUGH_PROPERTIES_HTC;
            unsafe
            {
                systemProperties.next = (IntPtr)(&SystemPassThroughPropertiesHTC);
            }
            int res = xrGetSystemProperties(ref systemProperties);
            if (res != (int)XrResult.XR_SUCCESS)
            {
                Debug.Log("Failed to get systemproperties with error code : " + res);

            }
        }

        public void SetSceneComputeOrientedBoxBound(Transform transform, Vector3 extent)
        {
            Vector4 rotation;
            Vector3 position;
            ConvertTransform(transform, out rotation, out position);
            NativeApi.SetSceneComputeOrientedBoxBound(rotation, position, extent);
        }

        private void ConvertTransform(Transform transform, out Vector4 rotation, out Vector3 position)
        {
            // Get left-handed values.
            position = transform.position;
            float angle;
            Vector3 axis;
            transform.rotation.ToAngleAxis(out angle, out axis);
            // Convert left-handed values to right-handed.
            position.z *= -1;
            angle *= -1;
            axis.z *= -1;
            var rotationQuaternion = Quaternion.AngleAxis(angle, axis);
            rotation = Vector4.zero;
            for (var i = 0; i < 4; ++i)
                rotation[i] = rotationQuaternion[i];
        }

        private class NativeApi
        {
            const string dll_path = "MeshingFeaturePlugin";
            [DllImport(dll_path)]
            public static extern void SetOpenXRVariables(ulong instance, ulong session,
                IntPtr PFN_XrEnumerateReferenceSpaces,
                IntPtr PFN_XrCreateReferenceSpace,
                IntPtr PFN_XrDestroySpace,
                IntPtr PFN_XrEnumerateSceneComputeFeaturesMSFT,
                IntPtr PFN_XrCreateSceneObserverMSFT,
                IntPtr PFN_XrDestroySceneObserverMSFT,
                IntPtr PFN_XrCreateSceneMSFT,
                IntPtr PFN_XrDestroySceneMSFT,
                IntPtr PFN_XrComputeNewSceneMSFT,
                IntPtr PFN_XrGetSceneComputeStateMSFT,
                IntPtr PFN_XrGetSceneComponentsMSFT,
                IntPtr PFN_XrLocateSceneComponentsMSFT,
                IntPtr PFN_XrGetSceneMeshBuffersMSFT);
            [DllImport(dll_path)]
            public static extern void SetSceneComputeOrientedBoxBound(Vector4 rotation, Vector3 position, Vector3 extent);
        }
    }
}

 

And here's the script for MeshingBehaviour which I named SceneUnderstanding_Sample

using VIVE.SceneUnderstanding;

using System.Collections;
using System.Collections.Generic;

using UnityEngine;
using UnityEngine.XR;
using UnityEngine.XR.OpenXR;
using UnityEngine.XR.OpenXR.Samples.MeshingFeature;
using UnityEngine.SubsystemsImplementation;
using UnityEngine.InputSystem;


public class SceneUnderstanding_Sample : MonoBehaviour
{
    public GameObject emptyMeshPrefab_Default;
    public Transform target;
    private XRMeshSubsystem s_MeshSubsystem;
    private List<MeshInfo> s_MeshInfos = new List<MeshInfo>();
    private Dictionary<MeshId, GameObject> m_MeshIdToGo = new Dictionary<MeshId, GameObject>();
    private MeshingTeapotFeature m_MeshingFeature;
    //Scene compute bound variables
    // A default cube game object
    public GameObject m_BoxBoundObject;

    // Start is called before the first frame update
    void Start()
    {
        m_MeshingFeature = OpenXRSettings.Instance.GetFeature<MeshingTeapotFeature>();
        Debug.Log(m_MeshingFeature.name);
        if (m_MeshingFeature == null || m_MeshingFeature.enabled == false)
        {
            enabled = false;
            return;
        }

        var meshSubsystems = new List<XRMeshSubsystem>();
        SubsystemManager.GetSubsystems(meshSubsystems);
        if (meshSubsystems.Count == 1)
        {
            s_MeshSubsystem = meshSubsystems[0];
            // textMesh.gameObject.SetActive(false);
        }
        else
        {
            enabled = false;
        }
    }

    // Update is called once per frame
    void Update()
    {
        if (m_BoxBoundObject == null) return;
            m_MeshingFeature.SetSceneComputeOrientedBoxBound(m_BoxBoundObject.transform, m_BoxBoundObject.transform.localScale); // The widths of a default cube is 1.0f.

        if (s_MeshSubsystem.running && s_MeshSubsystem.TryGetMeshInfos(s_MeshInfos))
        {
            foreach (var meshInfo in s_MeshInfos)
            {
                switch (meshInfo.ChangeState)
                {
                    case MeshChangeState.Added:
                    case MeshChangeState.Updated:
                        if (!m_MeshIdToGo.TryGetValue(meshInfo.MeshId, out var go))
                        {
                            go = Instantiate(emptyMeshPrefab_Default, target, false);
                            m_MeshIdToGo[meshInfo.MeshId] = go;
                        }

                        var mesh = go.GetComponent<MeshFilter>().mesh;
                        var col = go.GetComponent<MeshCollider>();

                        s_MeshSubsystem.GenerateMeshAsync(meshInfo.MeshId, mesh, col, MeshVertexAttributes.None,
                            result =>
                            {
                                result.Mesh.RecalculateNormals();
                            });
                        break;
                    case MeshChangeState.Removed:
                        if (m_MeshIdToGo.TryGetValue(meshInfo.MeshId, out var meshGo))
                        {
                            Destroy(meshGo);
                            m_MeshIdToGo.Remove(meshInfo.MeshId);
                        }
                        break;
                    case MeshChangeState.Unchanged:
                        break;
                    default:
                        break;
                }
            }
        }
    }
}

 

P.S. I also tried the example for SRWorks and I also can't get it to Play.

[KyleC]

Hi @maxxsby,

Firstly, we would like to check the environment on your PC where we can make the SRWork run successfully by following trouble-shooting:

1. Which headset were you using? If it was VIVE Pro, please make sure the camera is enabled as below.

 

2. When you play SRWork sample or OpenXR scene understanding, does SRWork runtime show in the system tray as below?

If no, could you share what installation you've made to install our VIVE SW? Is it from VIVE official installer or from steam App?

 

3. If the above two items are all correct, please help to provide the logs for us to analyze it furthermore.

=> Right click on system tray of SRWork runtime and then "Pack log".

[maxxsby]

For Number 1, I already did set the settings for the Camera and also the developer function for OpenXR. 

I am using VivePro2 and I also already tried running it with OpenXR without your samples and only the XR Rig which is the default VR package for Unity and it works fine. But not for any of your samples.

Here's the Camera Settings 

Here's the Developer Settings

As for step Number 2, the SRWork Runtime does not appear in my Taskbar. Your tutorial also did not mention any apps other than vive console to install. So, I was wondering if there are any other tutorial that I am missing, because from what you have told me I definitely have a step that I didn't do which definitely I didn't see in the tutorial.

And also regarding my original question, which is the script of MeshingBehaviour.cs from your sample or in my project is called SceneUnderstanding_Sample.cs still kept disabling itself be it from your sample or creating my own through your tutorial. As you can see from here.

 

Regarding your question for Vive Console installer. I tried two of your installation from Online and Offline Vive setup from your website. and yes, none of the samples I mentioned works in any of them.

Edited February 22, 2023 by maxxsby

[maxxsby]

Finally after some bits of tweaking, the SRWorks is working. But it still have some bugs. First of all you cannot install any other VR package other than com.unity.xr.management and xrcom.unity.interaction.toolkit. Even if it is from comhtc.upm.vive.openxr, because it will make the vive settings not working and you have to make another new project. Secondly my bug was, when I try to change the room view, the passthrough doesn't work anymore and it only display blank screen on Unity. Thirdly my Scene Understanding is still not working

[maxxsby]

@KyleC Hi Kyle, haven't heard from you. Is there a solution for the problem for SceneUnderstanding I'm currently having?? And do you know what happened for my camera at SRWorks to go blank?

[KyleC]

@maxxsbyWe are trying to reproduce the issue on our side and figure it out. Will get back to you soon.

[maxxsby]

@KyleC If you want to recreate it, here's my steps using Unity 2021.3.2f1:
Scene Understanding:
1. Package Manager install com.unity.xr.management and com.unity.xr.interaction.toolkit
2. Package Manager register vive registries:
VIVE
http://npm-registry.vive.com
com.htc.upm
3. Package Manager install com.htc.upm.vive.openxr and import SceneUnderstanding sample or make custom SceneUnderstanding via your tutorial (which actually have an error which is ConvertTransform where you never put it in your tutorial)
4. Play the scene.

For me whenever I play the scene it will always make the MeshBehaviour.cs disabled like the picture on my previous reply and also sometimes crashes when I turn on the vive console.

SRWorks:

1. Package Manager install com.unity.xr.management and com.unity.xr.interaction.toolkit

2. Import srworks package *IMPORTANT: DO NOT IMPORT STEAM VR OR VIVE OPENXR, because it will made you cannot accept the settings provided by srworks package when you change your scene. 

3. Play

For me when I play the scene it displays the camera passthrough on the game screen, but it will display nothing on the vive pro 2. The game screen on Unity also went blank when I tried to change the room view setting on SteamVR, and whenever I tried to play the scene again it will still be blank regardless if I try to change my vr room view setting back on the previous one again.

Edited February 23, 2023 by maxxsby

[KyleC]

@maxxsby

Thanks for sharing the reproduce steps. We've tried to follow the steps to reproduce the issue but with no luck.

But one suspect point is did you get an pop out window like below to tell you to restart the editor when you imported the scene understanding sample. And then click on Yes.

However, I think you might have done it right.

Secondly, could you try to create an new project but only do step 2 to step 4 to make sure eveything is new.

At the same time, check the srwork runtime is in the system tray.

 

Besides, here are the versions of each package when we tried to reproduce the issue:

• com.unity.xr.management : 4.2.1
• com.unity.xr.interaction.toolkit : 2.0.4
• com.htc.upm.vive.openxr : 1.0.10

 

 

[maxxsby]

@Kyle C Ok now it is enabled but it is still not like your picture in tutorial and neither the passthrough are working. Do I have to set another setting for the camera? Oh yeah, and I tried to get the log from pack log and it results in nothing, do I need to select the folder the project is on or select an empty folder?

Edited February 24, 2023 by maxxsby

[KyleC]

@maxxsby Glad to hear good news from you.

To show the mesh, please connect a controller and press "grip" key.

As to the passthrough, currently we only support to show it on our Cosmos HMD.

For VIVE Pro series HMD which use SteamVR runtime, you might ask Valve that how to enable/open passthrough in OpenXR.

 

As I know, Unity does not open environment blend mode API in OpenXR (which is used for AR mode) for developers, so it is hard for you to enable passthrough through OpenXR interface for now.