Enhancement of RenderTargetBitmap to Support Hardware Rendering

[DearVa]

Description

In WPF, RenderTargetBitmap (also RTB) currently only supports software rendering. This limitation leads to performance bottlenecks and causes ShaderEffect implementations to become ineffective. Enhancing RTB to support hardware rendering would provide significant performance improvements and enable high-quality rendering features that are currently restricted. This limitation has been extensively discussed in the past (#144, Add optional hardware acceleration to RenderTargetBitmap and some website was lost: RenderTargetBitmap and Hardware WPF Rendering).

As a developer who is proficient in graphics and C#, one of my requirements is to export WPF-rendered UI as video efficiently. Current methods such as using BitmapCache (which cannot retrieve content back to memory) and hooking DirectX APIs (which is complex and unstable) are not feasible solutions.

Technical Investigation

Upon inspecting the WPF source code in api_factory.cpp, I found that the creation of RenderTargetBitmap is restricted to software rendering:

STDMETHODIMP
CMILFactory::CreateBitmapRenderTarget(
    UINT width,
    UINT height,
    MilPixelFormat::Enum format,
    FLOAT dpiX,
    FLOAT dpiY,
    MilRTInitialization::Flags dwFlags,
    __deref_out_ecount(1) IMILRenderTargetBitmap **ppIRenderTargetBitmap
    )
{
    // ...
    
    if (SUCCEEDED(hr))
    {
        if ( !(dwFlags & MilRTInitialization::HardwareOnly) )
        {
            MIL_THR(CSwRenderTargetBitmap::Create(
                width,
                height,
                format,
                dpiX,
                dpiY,
                DisplayId::None,
                ppIRenderTargetBitmap
                DBG_STEP_RENDERING_COMMA_PARAM(NULL) // pDisplayRTParent
                ));
        }
        else
        {
            MIL_THR(WGXERR_NOTIMPLEMENTED);
        }
    }

    // ...
}

Proposed Modification for Hardware Rendering

To enable hardware rendering, I modified the above code as follows:

if (SUCCEEDED(hr))
{
    HWND const hwnd = GetDesktopWindow();  // TODO: get D3D Device is better
    CDisplaySet const *pDisplaySet = NULL; 
    MIL_THR(GetCurrentDisplaySet(&pDisplaySet));

    CDisplay const *pDisplay = pDisplaySet->Display(0);  // TODO: multi-display?
    CHwDisplayRenderTarget *pRenderTarget = NULL;
    MIL_THR(CHwDisplayRenderTarget::Create(
        hwnd,
        MilWindowLayerType::NotLayered,
        pDisplay,
        D3DDEVTYPE_HAL,
        dwFlags,
        OUT &pRenderTarget));

    IntermediateRTUsage rtUsage;
    rtUsage.flags = IntermediateRTUsage::ForBlending;
    rtUsage.wrapMode = MilBitmapWrapMode::Extend;
    MIL_THR(pRenderTarget->CreateRenderTargetBitmap(
            width,
            height,
            rtUsage,
            dwFlags,
            ppIRenderTargetBitmap));
}

Results and Observations

After compiling wpfgfx_cor3.dll with the modifications, WPF operates normally and, for the first time, provides high-performance rendering with support for ShaderEffect. I successfully converted the output to a WriteableBitmap and encoded it as a JPEG image. I am excited to share this successful update and would like to submit a PR to address this issue.

Concerns and Considerations

One potential reason why this approach might not have been implemented officially could be due to HwRenderTarget requiring a context (D3DDevice) binding. Since RenderTargetBitmap was developed later, it doesn't fit into DUCE and requires separate API calls, where the device context isn't readily available.

Proposed Solution

I plan to implement a new class derived from CHwSurfaceRenderTarget. This class will attempt to access the associated hardware m_pD3DDevice at creation (as WPF consistently uses a single D3DDevice). If unavailable, it defaults to CD3DDeviceManager::GetSWDevice. The new class's CreateRenderTargetBitmap will handle the selection of the correct RTB automatically. However, it raises key issues such as how to consistently access WPF's unique m_pD3DDevice and handle device loss (e.g., due to graphics driver updates).

I look forward to your feedback on this proposal.

[DearVa]

To further demonstrate the necessity, I ran a benchmark of RenderTargetBitmap.Render with a resolution of 1920x1080, with 197 Controls in total.
System.Windows.Controls.Border: 35
EasyPathology.Desktop.Replay.Views.Controls.ReplayViewContainer: 1
System.Windows.Controls.Grid: 30
EasyPathology.Desktop.Replay.Views.Controls.ReplayView: 1
EasyPathology.Desktop.Core.Views.Controls.SimplePanel: 6
EasyPathology.Desktop.Slide.Views.Controls.SlideRenderer: 1
System.Windows.Controls.TextBlock: 22
EasyPathology.Desktop.Core.Views.Controls.DrawingVisualElement: 4
System.Windows.Media.DrawingVisual: 4
EasyPathology.Desktop.Replay.Views.Controls.HeatMapRenderer: 1
EasyPathology.Desktop.Replay.Views.Controls.ReplayRecordableLayer: 1
EasyPathology.Desktop.Slide.Views.Controls.SlideCommentControl: 1
EasyPathology.Desktop.Core.Views.Controls.OffsetBindableScrollViewer: 2
System.Windows.Controls.TextBox: 1
Wpf.Ui.Controls.Button: 6
EasyPathology.Desktop.Slide.Views.Controls.SlideToolsControl: 1
System.Windows.Controls.StackPanel: 7
System.Windows.Controls.Primitives.ToggleButton: 7
EasyPathology.Desktop.Slide.Views.Controls.SlideRulerToolControl: 1
EasyPathology.Desktop.Slide.Views.Controls.SlideMarkPenToolControl: 1
EasyPathology.Desktop.Replay.Views.Controls.GazePointRenderer: 1
System.Windows.Controls.Menu: 2
System.Windows.Controls.MenuItem: 3
Wpf.Ui.Controls.ToggleButton: 11
Wpf.Ui.Controls.SymbolIcon: 16
System.Windows.Controls.ContentPresenter: 28
Wpf.Ui.Controls.MenuItem: 1
System.Windows.Controls.Primitives.Popup: 2

With hardware accl:
Max: 15.8619ms
Min: 6.5098ms
Avg: 13.19208ms

Original:
Max: 108.2391ms
Min: 83.7178ms
Avg: 93.02613000000001ms

[lindexi]

Look forward to your optimization