Add support for the HMD's hidden area mesh

[mrbelowski]

DrawHiddenAreaMesh_RenderThread_index / DrawHiddenAreaMesh_index appears in the UESDK's IHeadMountedDisplayVT - is it possible to hook this and pass the HMD-provided hidden area mesh (or a derivative of it that accounts for projection changes)?

if not, could a hacky alternative be to create and attach an opaque mesh to the camera at the near-z distance which just occludes stuff in that part of the view?

[praydog]

Yes, but the answer is more complicated than that:

virtual void DrawHiddenAreaMesh_RenderThread(class FRHICommandList& RHICmdList, EStereoscopicPass StereoPass) const {};
virtual void DrawVisibleAreaMesh_RenderThread(class FRHICommandList& RHICmdList, EStereoscopicPass StereoPass) const {};

This expects the implementor (the VR plugin AKA UEVR) to call the relevant functions at this point in the rendering pipeline to draw the mesh primitives, given the command list.

Adding a command to the command list properly would need to be reverse engineered, and tested for stability across versions. We would use something like the engine function DrawIndexedPrimitiveUP with the mesh layout to do this. There is a lot of moving parts in this which makes me kind of skeptical of its version stability.

So these are the main problems:

1. If the DrawIndexedPrimitiveUP function is not inlined, we would need to construct a scan routine to locate the function. I am very against signatures/patterns as they are hard to maintain, not human readable, prone to breaking. There are tons of examples of how I approach this "limitation" I have imposed within the SDK
2. Constructing the vertices in the format UE expects and passing it to this function

The other alternative would be constructing our own injected DirectX command which would arguably be easier to maintain, into the correct point within the rendering pipeline. If done directly through the engine, the DirectX implementation for 11 and 12 could be abstracted away and would be preferable.

According the the source, DrawIndexedPrimitiveUP is declared as inline so this already presents a problem. The functionality inside of it would need to be recreated.

[praydog]

If FRHICommandEndDrawIndexedPrimitiveUP's vtable is in every game, and its layout is the same, that's easily recreatable. Allocate a new structure with the vtable pointing towards this, and assume the structure layout is the same, recreate the constructor, then allocate the command in the list.

The RHICmdList layout is relatively simple near the start, so this could be allocated. The DrawUPData offset within the FRHICommandListBase would just need to be located which is a bigger issue as it's deeper within the structure.

[praydog]

Now, as an alternative, we could instead hook the last command's Execute function (it's a virtual function) in the RHICmdList at the time DrawHiddenAreaMesh_RenderThread is called and call the IRHICommandContext's RHIBeginDrawIndexedPrimitiveUP and RHIEndDrawIndexedPrimitiveUP. This is dependent on knowing the virtual indices within the context for these. This would work because the pipeline state would already be set up at the time it's called.

[praydog]

An easy experiment would be to hardcode all the necessary offsets and see if any of these methods (apart from the DX one) works.

Well this was a lot of brainstorming. Good documentation for later.

[mrbelowski]

great documentation for later and some really clear explanations as to why this is so challenging. I really appreciate the time you've taken to explore and explain this.

Obviously all of the stuff above is way beyond my knowledge and skill level, and the chances of me implementing something workable are basically zero, but I'll take a look anyway because it's fun

[praydog]

I have been writing (and learning...) some Direct3D code in some unfinished builds, so I might be able to experiment with obtaining the actual D3D command list at the time this virtual is called, and attempt to inject an actual Direct3D DrawIndexedPrimitiveUP-like command into the depth buffer (at least that's what I assume is supposed to be done to make this work) if nothing fancy is needed