Update GS rendering #6352

slimbuck · 2024-05-10T15:38:36Z

This PR updates our runtime GS renderer. GPU render performance is improved as well as sorting performance and jank during camera movement.

Before:

After:

The code changes are summarised as follows:

at load time we reorder splat data to be gpu cache friendly (using morton order)
- this is skipped when loading the compressed PLY format because data is already ordered
the cpu sorter writes results to a texture and calculates number of splats in front of the camera
render with a two phase approach:
- phase1: (preprocess) transform splat data into camera space - store the resulting data in render order (this is only required when camera moves)
- phase2: (render) consume preprocess data and render quads

Other changes:

splats are rendered in batches of 128 quad mesh instances to improve occupancy

Issues:

the load time reordering is time and memory intensive. we may want to perform this on a worker.
this technique uses more gpu memory than before (to store the phase1 output). this might be an issue for mobile devices.
phase1 output stores splat XYZW to a RGBA16F buffer. This appears to be enough precision, but more testing is needed to confirm.
a followup PR will restructure shaders to accommodate the rendering customisations we need in, for example SuperSplat
VR mode may require work. This will be a followup PR.
The texture upload API doesn't support uploading partial data, so we can't upload only the required sorting data.

TODO:

implement rendering directly from compressed format (we currently decompress the data at load time). this should result in good memory and performance speedup and should fit nicely into the existing rendering pipeline.
investigate fragment bandwidth optimisation by:
- switch to rendering the splats front to back (instead of back to front as we do now)
- render half the splats
- stencil out/mark pixels that are fully opaque
- render second half of the scene skipping already filled pixels

This reverts commit eff37ce.

mvaligursky · 2024-05-10T15:57:04Z

What's the impact to WebXR rendering using two viewports?

Maksims · 2024-05-10T18:36:25Z

Is it possible to configure size of batches? As in case of 1m splats, it will lead to ~7.8k drawcalls?

src/scene/gsplat/gsplat-data.js

willeastcott · 2024-05-10T21:55:29Z

src/scene/gsplat/gsplat-instance.js

+    centerTexture;
+
+    colorTexture;
+
+    v1v2RenderTarget;
+
+    v1v2RenderPass;


willeastcott · 2024-05-10T21:58:47Z

src/scene/gsplat/gsplat-instance.js

@@ -120,6 +307,10 @@ class GSplatInstance {

    createMaterial(options) {
        this.material = createGSplatMaterial(options);
+        // this.material.setParameter('splatOrder', this.orderTexture);


willeastcott

A true work of art. 🖼️

willeastcott · 2024-05-10T22:15:22Z

@slimbuck Would it be reasonable for this PR to close #6344?

slimbuck · 2024-05-11T14:58:31Z

Is it possible to configure size of batches? As in case of 1m splats, it will lead to ~7.8k drawcalls?

@Maksims, the 128-quad mesh is instanced as many times as necessary with single draw call.

slimbuck · 2024-05-11T14:59:21Z

What's the impact to WebXR rendering using two viewports?

Yeah thanks I forgot to mention in the issues section. Plan is to fix/address VR in a followup PR.

slimbuck · 2024-05-11T15:00:37Z

@slimbuck Would it be reasonable for this PR to close #6344?

Hopefully! :D

mvaligursky · 2024-05-13T08:49:52Z

src/scene/gsplat/gsplat-instance.js

+        scope.resolve('transformB').setValue(this.splat.transformBTexture);
+        scope.resolve('transformC').setValue(this.splat.transformCTexture);
+
+        const cameraMatrix = new Mat4();


these seem like per frame allocations?

slimbuck · 2024-05-13T14:06:47Z

Closing in favour of #6357.

slimbuck added 30 commits April 25, 2024 10:57

move render order to texture

4a02351

remove unneccessary vertex data

142b50d

remove dead code

ea02fbf

small

a3aa33b

create correct size target

5871e3a

fix for firefox - specify per-splat indices

4061ff7

sort by camera distance

eff37ce

preprocess v1v2 texture

144bacf

using instancing

f135b23

Merge branch 'main' into gs-sorter-v1v2-instanced

2daf06b

Revert "sort by camera distance"

ebe5f8d

This reverts commit eff37ce.

small

514122e

use sort tex

13e8c09

small

c9a3806

small

1ae4465

bug

0bf88db

preprocess in v1v2

a49d351

only preprocess after sort

f46cdae

Merge branch 'main' into gs-sorter-v1v2-instanced-sorttex-preprocess

5b6a725

revert unused changes

9a88801

more

854e7c2

small

5ff81f7

Merge branch 'main' into gs-sorter-v1v2-instanced-sorttex-preprocess

060c8bc

Merge branch 'main' into gs-sorter-v1v2-instanced-sorttex-preprocess

38a7e4d

small

18ac422

working on android

f9efc39

small

61ddce8

reorder splat data at load

8e31df6

use render pass

157a96a

simplify pass

1e779f9

slimbuck added 4 commits May 9, 2024 17:47

small

512369e

fix splat culling

5f1dce2

render last instance

030ab27

Merge branch 'main' into gs-sorter-v1v2-instanced-sorttex-preprocess

1751dda

slimbuck added area: graphics Graphics related issue enhancement labels May 10, 2024

slimbuck requested a review from a team May 10, 2024 15:38

slimbuck self-assigned this May 10, 2024

lint

3485f04

mvaligursky mentioned this pull request May 10, 2024

Improve Gaussian Splats performance #6344

Open

willeastcott reviewed May 10, 2024

View reviewed changes

src/scene/gsplat/gsplat-data.js Outdated Show resolved Hide resolved

willeastcott reviewed May 10, 2024

View reviewed changes

willeastcott approved these changes May 10, 2024

View reviewed changes

mvaligursky reviewed May 13, 2024

View reviewed changes

mvaligursky approved these changes May 13, 2024

View reviewed changes

slimbuck added 2 commits May 13, 2024 10:42

changes based on feedback

4c4c234

Merge branch 'main' into gs-sorter-v1v2-instanced-sorttex-preprocess

7b90dd5

slimbuck mentioned this pull request May 13, 2024

Update GS rendering v2.0 #6357

Merged

slimbuck closed this May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update GS rendering #6352

Update GS rendering #6352

slimbuck commented May 10, 2024 •

edited

mvaligursky commented May 10, 2024

Maksims commented May 10, 2024

willeastcott May 10, 2024

willeastcott May 10, 2024

willeastcott left a comment

willeastcott commented May 10, 2024

slimbuck commented May 11, 2024

slimbuck commented May 11, 2024

slimbuck commented May 11, 2024

mvaligursky May 13, 2024

slimbuck commented May 13, 2024

Update GS rendering #6352

Update GS rendering #6352

Conversation

slimbuck commented May 10, 2024 • edited

mvaligursky commented May 10, 2024

Maksims commented May 10, 2024

willeastcott May 10, 2024

Choose a reason for hiding this comment

willeastcott May 10, 2024

Choose a reason for hiding this comment

willeastcott left a comment

Choose a reason for hiding this comment

willeastcott commented May 10, 2024

slimbuck commented May 11, 2024

slimbuck commented May 11, 2024

slimbuck commented May 11, 2024

mvaligursky May 13, 2024

Choose a reason for hiding this comment

slimbuck commented May 13, 2024

slimbuck commented May 10, 2024 •

edited