HoloLens 2 Mixed Reality Capture's viewfinder not showing

[austinbhale]

Description

To reproduce, restart the device. Open a StereoKit app, and say "Start Recording". Starting in October 2023, a viewfinder for the HoloLens 2 should appear when recording. It isn't until you exit the SK app that you'll be able to see the viewfinder when starting another recording or MRC session.

A strange tidbit is that if you start a recording outside of the SK app, the viewfinder appears as expected. But, if you jump back into the SK app and try to reproduce with a new recording, the viewfinder will appear. That's why I recommend restarting the device in order to reproduce.

Platform / Environment

HoloLens 2 UWP. Tested in v0.3.6 -> v0.3.9

Logs or exception details

I was able to get the std::runtime_error once after a fresh device restart, but it does not always occur. The rest of the log is consistent for successive runs of the app. The holographic space/camera exceptions appear when you stop recording, both when the frustum is and is not visible.

D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x141725D4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x14014F34, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5F2C4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5F5E4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5DCE4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5E964, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5EFA4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C5F904, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C60264, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1400C3AC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404D42C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404D23C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404D9FC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x140279AC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x14028C6C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x1402704C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C62E24, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C627E4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C62B04, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404E3AC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404E78C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404DBEC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404DDDC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x140286CC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x14027F4C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x140288AC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C60BC4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C61E84, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11Texture2D: Name="unnamed", Addr=0x17C63144, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097234: CREATE_TEXTURE2D]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404E97C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404EB6C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x1404ED5C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x163DC23C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x1402830C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x140284EC, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11UnorderedAccessView: Name="unnamed", Addr=0x14009F4C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097301: CREATE_UNORDEREDACCESSVIEW]
D3D11 INFO: Create ID3D11Buffer: Name="unnamed", Addr=0x17C60584, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097228: CREATE_BUFFER]
D3D11 INFO: Create ID3D11Buffer: Name="unnamed", Addr=0x17C60EE4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097228: CREATE_BUFFER]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x163BDAA4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11RenderTargetView: Name="unnamed", Addr=0x1404144C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097243: CREATE_RENDERTARGETVIEW]
D3D11 INFO: Create ID3D11DepthStencilView: Name="unnamed", Addr=0x163BDEE4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097246: CREATE_DEPTHSTENCILVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x163C0964, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11RenderTargetView: Name="unnamed", Addr=0x1404164C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097243: CREATE_RENDERTARGETVIEW]
D3D11 INFO: Create ID3D11DepthStencilView: Name="unnamed", Addr=0x163C0B84, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097246: CREATE_DEPTHSTENCILVIEW]
D3D11 INFO: Create ID3D11ShaderResourceView: Name="unnamed", Addr=0x163C1624, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097240: CREATE_SHADERRESOURCEVIEW]
D3D11 INFO: Create ID3D11RenderTargetView: Name="unnamed", Addr=0x0F0E924C, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097243: CREATE_RENDERTARGETVIEW]
D3D11 INFO: Create ID3D11DepthStencilView: Name="unnamed", Addr=0x163BFEC4, ExtRef=1, IntRef=0 [ STATE_CREATION INFO #2097246: CREATE_DEPTHSTENCILVIEW]
Exception thrown at 0x776C6C7F in NARA.UWP.exe: Microsoft C++ exception: std::runtime_error at memory location 0x0537DE18.
[SK info] xrEndFrame [XR_ERROR_RUNTIME_FAILURE]
analog\input\mirage\publicapi\holographicspace.cpp(346)\Windows.Mirage.dll!70972881: (caller: 709FAD3B) ReturnHr(2) tid(15cc) 80004005 Unspecified error
analog\input\mirage\publicapi\holographiccamera.cpp(208)\Windows.Mirage.dll!709FAD4D: (caller: 709FCBBB) ReturnHr(3) tid(15cc) 80004005 Unspecified error

[austinbhale]

At times, we've also noticed that when MRC is activated in an SK app, it will crash unexpectedly. Closing the app or restarting usually resolves it. Unfortunately, I don't have any additional logs or debugging info yet

[maluoi]

I'm about to wrap up some work on a tricky feature, but after that I think I owe UWP (like #981) a bit of time and attention. I'll definitely be looking into this next week!

[austinbhale]

Sweeet, thanks so much :)

[laultman]

Likewise, thanks!

[austinbhale]

On the version/0.3.10 branch using the following repro strategy:

1. Restart the HoloLens 2 (every time!)
2. Open the StereoKitTest_UWP app
3. Start a MRC by saying the "Start Recording" HL2 command

I am still getting the error, but there's some hope! By commenting out the primary display's OpenXR composition layer when the secondary display is active, it successfully shows the viewfinder again but obviously nothing in StereoKit is rendered :D so I'm a little stuck here at the moment

Another quirk is if you're debugging it and set a breakpoint with the branch's code, it'll actually show up as it should so maybe SK is holding onto something with the primary display?

[maluoi]

Is this specifically about the viewfinder corners? Not a crash of any sort? I was able to see the corners not showing up the first time after restart, but I haven't seen any functional impact beyond that. The recording camera icon was still visible there too!

By commenting out the primary display's OpenXR composition layer when the secondary display is active, it successfully shows the viewfinder again but obviously nothing in StereoKit is rendered

I wonder if this is maybe a composition layer order issue? Maybe HL2's viewfinder composition layer sorts itself underneath SK's projection layer in some cases. If that is what's happening, then this might be very tricky to solve on our end. We don't really have control over ordering of composition layers outside of our own.

[austinbhale]

This is mainly about the viewfinder corners! I haven't been able to repro a crash anymore. The performance in the StereoKit test app definitely looks normal (stable 30 FPS with MRC enabled). I tried the BasicXrApp in https://github.com/microsoft/OpenXR-MixedReality and it had the same issue, so maybe I'll drop it there. Perhaps it's an OpenXR-related bug since non OpenXR apps don't have this issue

[maluoi]

Okay, if that project also reproduces it, I'm not feeling so bad about this 😅 I guess this is not currently a blocker or anything at this point?

I did find/fix a couple sketchy code-paths with the secondary view when re-writing the swapchain code to support MSAA! I'm not so sure they'd have been common problems (or problems at all), but the new code does bullet-proof that feature a fair bit more than before. If you catch any hints of problems still coming from there with v0.3.10 though, lemme know!

[austinbhale]

Nope, not a blocker! Thanks for fixing up those paths, it's much appreciated :) They'll get some traffic as I update to the new version

I'll close this one and make an issue over there because I repro'd in their OpenXR Tools official sample too with the latest runtime