Add a configurable Fog effect #10564

karimnaaji · 2021-04-13T18:45:11Z

This PR adds configurable fog to the style specification. Mapbox-gl maps use draw order and hues as a way to prioritize categories of layers. In 3D (e.g. high pitch or with terrain), visual hierarchies may be lost due to arbitrary viewing angles. As a result, the position of objects on the vertical axis becomes our main visual hint to understanding where objects fall at the distance.

The fog effect helps to smooth the horizon line and gives a better understanding of the scene layout in 3D, it is an important visual clue as we are used to evaluating the distance of objects in a landscape by how blue they appear in relation to each other. In addition to visual features, fog can be used to reduce the perceived scene as it allows for a fog tile culling optimization that reduces tile count fetches, as the viewable bounds can be set closer to the viewer.

It is implemented with the following considerations:

The fog effect is applied at a very late stage of the shader processing pipeline for all layers except terrain (forward shading).
- For terrain: some of the effect evaluation is done in the vertex shader. As the mesh is known to be highly tesselated, moving some operations from fragment to vertex shader resulted in a measurable performance improvement (~1ms per GPU frame time on a gt3e at 1080p) and very little visual difference.
- For every other layer, it is evaluated in the fragment shader of the respective layer.
- @arindam1993 evaluated a post-processing pipeline (rendering the fog as a deferred render pass). Although the simplicity of this effect was appealing, it resulted in a few issues so we did not pursue:
  - Loss of granularity of how the fog effect should be applied. Our renderer may sometimes use premultiplied alpha where fog needs to be applied slightly differently.
  - As we would be sampling a depth buffer, label billboards may evaluate very different fog factors within the same quad (Refer screenshot below).
  - The depth buffer that we currently create for terrain would not be useable as-is for this effect. We would probably need a separate one that includes fill-extrusion for this to work.
  - As a requirement to support hardware lacking support of non-pot offscreen render target, using a depth resolution that was not matching the viewport resulted in various aliasing artifacts:
background.mov
Markers use a few different opacity stops at which they evaluate their opacity against fog. As markers are not rendered natively, directly updating their opacity could result in style re-layout and a performance slow-down. The tradeoff is to have a slightly less accurate opacity for a faster marker update loop.

markers-stops.mov

Tile fog culling is implemented at a very late stage of the fog tile cover after the initial cover finished defining the visible tileset with frustum culling. It can be used as a mean to improve styling performance when visible distance is not a strict requirement of the style.
- To have a slightly better culling result, we cull tiles at about 98% fog opacity, leaving a non-noticeable opacity change threshold.
- We cull by evaluating the fog opacity of the furthest tile corner in fog space and compare against the fog end range.
- Tiles cutting over the horizon line are rejected from the fog culling logic, as this could result in visible pop-in and pop-out when navigating the map at high pitch/zoom.
culling-range.1.mov
The fog is a root property and can be used with the following specification:
- range: The near and far limits of the fog layer. Units are relative to the map height, so that 2.0 corresponds to twice the map height, in pixels.
- color: The color of the fog. If provided as a rgba color, the alpha component is unused.
- strength: Controls the strength of the fog effect. Reducing the fog strength pushes the fog farther toward the horizon, leaving more visible space for haze if enabled. When haze energy is zero, this will usually be 1.0.
- haze-color: The color of the haze component of the fog, used only when haze energy is nonzero. If provided as a rgba color, the alpha component is unused.
- haze-energy: Controls the strength of haze in the fog layer. Haze is an additive effect that approximates the look of an atmosphere. Increasing this strengthens the haze. Reducing the fog's strength pushes fog toward the horizon, making haze visible.
- sky-blend: The sky blend allows to blend the effect of the fog above the horizon line. A value of 1 will entirely blend the fog onto the sky, while a value of 0 will leave the sky unaffected by fog.
Symbols are automatically clipped on the CPU at a given fog opacity (currently 90%) to prevent showing labels without surrounding context. We are working on further improvements to labeling to have more granular control per feature for a given fog opacity. Doing it on the CPU has the advantage of preventing false positives compared to a GPU approach, as occluded labels would not hide other labels.

To prevent potential banding artifacts (as fog is a natural gradient), we apply a temporal dithering to the effect, exaggerated here for the sake of the visualization (without and with dithering):

Defaults:
- Fog is a no-op for the 2d use case (below < 60 pitch)
- The fog is not enabled by default to not introduce breaking changes to the library, it has very light and non-intrusive defaults, the result of map.setFog({}).

Optimization that were not implemented in this PR and left for further investigation:

Since tiles may be drawn a null fog/haze contribution (basically running all of the fragment operations that results in no fog contribution at all) we could evaluate the fog opacity of the furthest tile corner CPU side and check if it's < fog start range. We would switch the shader variant to use a non-fog shader (we can do that easily with the define). This could reduce fragment ops as these tiles are close to the viewer and have a lot of screen area/fragments to work upon. The shader switches would only happen once as tiles are already sorted by distance to the viewer. The tiles that would be subject to this switch are represented in blue here:

cc @mapbox/gl-native, @mapbox/map-design-team

Launch Checklist

briefly describe the changes in this PR
include before/after visuals or gifs if this PR includes visual changes
write tests for all new functionality
document any changes to public APIs
manually test the debug page
tagged @mapbox/map-design-team @mapbox/static-apis if this PR includes style spec API or visual changes
tagged @mapbox/gl-native if this PR includes shader changes or needs a native port
apply changelog label ('bug', 'feature', 'docs', etc) or use the label 'skip changelog'
add an entry inside this element for inclusion in the mapbox-gl-js changelog: <changelog>Add a configurable fog effect as a root style specification</changelog>

karimnaaji · 2021-04-16T23:41:43Z

Marking this PR as ready for review. There are still a few benchmarking job runs that I'd like to run against this PR from the bench suite but we're not expecting any more major changes to happen other than spec details and review changes. The work from @arindam1993 on symbols will be easier to digest as a separate PR.

asheemmamoowala · 2021-04-17T00:00:09Z

src/style-spec/reference/v8.json

+          "zoom"
+        ]
+      },
+      "doc": "The near and far limits of the fog layer. Units are relative to the map height, so that 2.0 corresponds to twice the map height, in pixels.",


Can this new unit type be named and defined so that it can be referenced uniformly across other style properties or in API docs as well

If we multiply by 100, it's CSS vh units, but otherwise I'm not sure the best name for it.

The near and far limits of the fog layer.

This suggests that fog only exists within the range. It might be good to document that this is the range that fog fades in over

src/ui/map.js

src/style/style.js

karimnaaji · 2021-04-20T22:41:47Z

Reference benchmark run of main fixture:

src/ui/marker.js

src/shaders/_prelude_fog.fragment.glsl

src/render/painter.js

src/geo/transform.js

src/render/painter.js

ansis · 2021-04-21T22:18:27Z

I think we need to revisit the start/end range units. With the current units the appearance of fog changes very significantly when you:

pitch the map, or
reduce the viewport height

Set the debug/fog.html page to these settings to see the problem. The problem is still visible with only haze and a less sharp cutoff but these settings make it easier to see in screenshots.

Pitching example

As you pitch fog seems to appear and then recede. Areas that are covered at medium pitches are not covered if you pitch slightly more or slightly less. These all look like pretty different amounts of fog to me.

Fog also looks significantly different with different map heights but I'm not sure what the behavior in that case should be.

rreusser · 2021-04-21T22:29:12Z

@ansis there's a possibility you're seeing something different, but to the best of my knowledge, a good share of what you're seeing results from us not knowing how far away the terrain is (alternatively, what the zoom level is). Near sea level, things work pretty well, e.g. http://localhost:9966/debug/fog.html#13.75/-44.65923/167.90228/-47.8/74.

At high elevation though, we don't have a reliable way of telling whether what's in front of the camera is a passing hill which shouldn't affect the range, or whether it's the dominant terrain. Or to look at it differently, if you pan, the zoom level changes sporadically as hills pass in front of the camera.

To avoid fog jumping all over the place, we shoot a ray to sea level. When you're at sea level, that's fine. When you're viewing high elevation terrain, that may be very far away, essentially making fog disappear.

I tried to average the local minimum aabb elevation of the loaded tiles (with a kernel to favor what's near the camera). It helped. At high elevations, the fog was pulled 3-4x closer. I might consider resurrecting that. The downside is that as you pan, new tiles load, and it's a bit jumpy. It would probably need to be eased in time. Also, a single low-elevation pixel (of which there are some exceptionally bizarre ones), affect fog dramatically.

rreusser · 2021-04-21T22:52:23Z

Also, @ansis, here's what I see for resizing the viewport at sea level, which seems to me to work pretty well.

resize.mp4

It seems reasonable to me at high elevation as well. Maybe not quite as good (I think due to the above sea-level-ray issue, which is also the main culprit when you change the pitch)

high-elevation.mp4

If we just deal in pixel units (instead of pixel units divided by viewport height), the fog range has a strong dependence on map height:

without-viewport-scaling.mp4

ansis · 2021-04-21T23:56:06Z

@ansis there's a possibility you're seeing something different, but to the best of my knowledge, a good share of what you're seeing results from us not knowing how far away the terrain is (alternatively, what the zoom level is). Near sea level, things

Yeah, it looks like that is a large part of it. This is something we need to fix or we need to approach this differently. Fog should look pretty consistent across different pitches and at different elevations.

karimnaaji · 2021-04-22T00:09:55Z

Yeah, it looks like that is a large part of it. This is something we need to fix or we need to approach this differently. Fog should look pretty consistent across different pitches and at different elevations.

While I don't disagree, I don't think this is a blocker. Opening this up for improvements in subsequent work chunks is probably the best approach to prevent blocking this PR from merging. As fog is an opt-in feature that has convenient default at far distance (mitigating that specific pitch issue), it would be a fairly subtle change accomodating for the closer range case, that I don't foresee being the most common case.

Lint

* Continuously adjust the fog scale by raycasting * Remove unused function * Fix linter errors * Name thresholds for clarity * Hard-code sampling pattern * Rearrange conditional * Adjust sampling by horizon line * Replace arbitrary constant with epsilon * Remove stray import * Simplify logic * Only calc fog matrices on avg elevation change * Fix cloning of transform * Fix bug in state management * Fix linter errors * Remove debug code * Move samples points inside function * Fix accidental call to calcMatrices * We still love you, flow * Clarify code * Add tests for EasedVariable * Add one more test

Make sure to un/post multiply alpha before/after applying sky gradient contribution, as color ramps are premultiplied-alpha colors. This prevents washed out colors when using transparency.

* Account for FOV in fog * Fix a typo * Fix flow errors * Fix a factor of two to make tests pass * Fix bad merge * Fix tests * Shift fog range zero to the map center * Fix debug * Fix ranges to account for fov * Remove unused variable

when trying to reason about them. As the units of the range are representing 3d distances on a sphere centered around camera center point, they can hardly be reasoned about in terms of window height.

* Fix race condition in fog terrain sampling * Use frame start time stamp * Update render test expectation

* Remove fog vh units * Fix floating point errors in test expectations * Fix geo test

* Skip unnecessary uniform code path when terrain is rendering to texture On a typical frame with terrain this was called ~200 times, where only 20 were necessary (~10%) * Fix invalid transition evaluation When `delay` was undefined but not `now`, properties.begin and properties.end were assigned incorrect values, leading to an immediate transition interpolation. This was noticed when testing transitions on fog. * Fix style.hasTransition() when fog is transitioning values * Add a fog demo for release

src/render/painter.js

src/ui/marker.js

src/ui/map.js

ansis · 2021-05-06T16:30:09Z

src/ui/map.js

@@ -2426,6 +2486,17 @@ class Map extends Camera {
        this._canvas.style.height = `${height}px`;
    }

+    _addMarker(marker: Marker) {
+        this._markers.push(marker);


It seems a bit messy to have some marker changes happen by the marker listening to map events and other changes to happen by the map calling methods directly on them.

But also don't have a clear suggestion here.

src/ui/map.js

rreusser

A couple small things, but I don't think I'd consider any of my feedback blocking. 👏🌁

src/render/painter.js

rreusser · 2021-05-06T17:07:31Z

src/shaders/_prelude.glsl

+float fog_horizon_blending(vec3 camera_dir) {
+    float t = max(0.0, camera_dir.z / u_fog_horizon_blend);
+    // Factor of 3 chosen to roughly match smoothstep.
+    // See: https://www.desmos.com/calculator/pub31lvshf


I imagine these desmos links as being a nice convenience but not essential to understanding the code, but maybe it'd be a good idea for me to screenshot them and add them to the github PR before this work is all water under the bridge.

rreusser · 2021-05-06T17:15:15Z

src/style-spec/reference/v8.json

+    "horizon-blend": {
+      "type": "number",
+      "property-type": "data-constant",
+      "default": 0.1,


Because of the fog horizon fix and just for aesthetics, it's advisable to increase this with a zoom expression as you increase the zoom. I hope that doesn't get lost by the time this makes its way to the designers.

src/ui/map.js

src/util/util.js

…ting it

need to expose it publicly

Co-authored-by: Ricky Reusser <rreusser@users.noreply.github.com>

ansis

Looks amazing to me!

karimnaaji · 2021-05-07T15:45:51Z

Will go ahead with merging in a bit, captured the last review comments as follow ups:

Evaluate whether throttling of marker opacity updates is needed
Unit tests for -transitions behaviors
Unit tests for average elevation sampling (+reorg to separate files)

Thanks for the multiple reviews @ansis and nice work on this @rreusser and @arindam1993 .

karimnaaji · 2021-05-07T16:42:40Z

Latest benchmark run:

karimnaaji linked an issue Apr 13, 2021 that may be closed by this pull request

Add configurable fog to enhance 3d scene depth perception #10205

Closed

karimnaaji force-pushed the fog-implementation branch 2 times, most recently from 72916b7 to e8f7d74 Compare April 16, 2021 20:15

karimnaaji mentioned this pull request Apr 16, 2021

Fog style spec changes #10590

Merged

karimnaaji requested a review from a team April 16, 2021 23:39

karimnaaji marked this pull request as ready for review April 16, 2021 23:39

asheemmamoowala reviewed Apr 17, 2021

View reviewed changes

src/style/style.js Show resolved Hide resolved

karimnaaji mentioned this pull request Apr 19, 2021

Bypass the DOM rendering queue if map is idle #10567

Merged

karimnaaji force-pushed the fog-implementation branch from 673559b to 7a5554a Compare April 20, 2021 21:53

ryanhamley added this to the v2.3 milestone Apr 21, 2021

ansis reviewed Apr 21, 2021

View reviewed changes

karimnaaji force-pushed the fog-implementation branch 2 times, most recently from d965d76 to 7db9f61 Compare April 23, 2021 23:06

karimnaaji added 3d 📐 feature 🍏 labels Apr 23, 2021

karimnaaji mentioned this pull request Apr 29, 2021

Switch fog from percentage window height to CSS vh units #10632

Merged

karimnaaji force-pushed the fog-implementation branch 3 times, most recently from 69bfce9 to 798728e Compare May 4, 2021 00:31

karimnaaji added 3 commits May 6, 2021 07:55

Add fog stylespec

8a0bb5c

Add fog type specification

6206937

Add fog stylespec validation

2abd5bf

Lint

karimnaaji and others added 12 commits May 6, 2021 08:01

Fixup consistent naming

7e71f38

Add symbol clipping render test

f97e723

Fix default per latest changes and add render tests

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

c1f54b4

Fix alpha composing of horizon blend with sky gradient (#10640)

4e5eb18

Make sure to un/post multiply alpha before/after applying sky gradient contribution, as color ramps are premultiplied-alpha colors. This prevents washed out colors when using transparency.

Account for FOV in fog (#10629)

600512b

* Account for FOV in fog * Fix a typo * Fix flow errors * Fix a factor of two to make tests pass * Fix bad merge * Fix tests * Shift fog range zero to the map center * Fix debug * Fix ranges to account for fov * Remove unused variable

Apply review feedback: Directly update marker style opacity

b74d90b

Update per latest discussion: Use nameless units to prevent confusion

336ae37

when trying to reason about them. As the units of the range are representing 3d distances on a sphere centered around camera center point, they can hardly be reasoned about in terms of window height.

Fixup naming consistency

e1db8ed

Fix race condition in fog terrain sampling (#10642)

839fa3a

* Fix race condition in fog terrain sampling * Use frame start time stamp * Update render test expectation

Remove fog vh units (#10648)

72fd595

* Remove fog vh units * Fix floating point errors in test expectations * Fix geo test

karimnaaji force-pushed the fog-implementation branch from 29f0846 to a0d6705 Compare May 6, 2021 15:09

ansis reviewed May 6, 2021

View reviewed changes

src/render/painter.js Show resolved Hide resolved

ansis reviewed May 6, 2021

View reviewed changes

rreusser approved these changes May 6, 2021

View reviewed changes

karimnaaji and others added 8 commits May 6, 2021 13:09

Directly use DOM queue rencently introduced by #10530

158dece

Apply review feedback: Directly clear the timer here instead of unset…

5e436a5

…ting it

Apply review feedback: Fix nits, mark queryFogOpacity private until we

e5abf75

need to expose it publicly

Update src/util/util.js

876da72

Co-authored-by: Ricky Reusser <rreusser@users.noreply.github.com>

Update src/util/util.js

de60d71

Co-authored-by: Ricky Reusser <rreusser@users.noreply.github.com>

Apply review feedback: Fix nits

5b2538b

Apply review feedback: Simplify matrix initialization

824fabd

make private methods private for docs

87017f0

ansis approved these changes May 7, 2021

View reviewed changes

karimnaaji merged commit 2df9f37 into main May 7, 2021

karimnaaji deleted the fog-implementation branch May 7, 2021 17:44

Eldemarkki mentioned this pull request Dec 7, 2024

Stop marker from flashing on initial load if it's occluded by the globe #13344

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a configurable Fog effect #10564

Add a configurable Fog effect #10564

karimnaaji commented Apr 13, 2021 •

edited

Loading

karimnaaji commented Apr 16, 2021

asheemmamoowala Apr 17, 2021

rreusser Apr 17, 2021

ansis Apr 27, 2021

karimnaaji commented Apr 20, 2021

ansis commented Apr 21, 2021

rreusser commented Apr 21, 2021 •

edited

Loading

rreusser commented Apr 21, 2021 •

edited

Loading

ansis commented Apr 21, 2021

karimnaaji commented Apr 22, 2021

ansis May 6, 2021

ansis May 6, 2021

rreusser left a comment

rreusser May 6, 2021

rreusser May 6, 2021 •

edited

Loading

ansis left a comment

karimnaaji commented May 7, 2021

karimnaaji commented May 7, 2021

Add a configurable Fog effect #10564

Add a configurable Fog effect #10564

Conversation

karimnaaji commented Apr 13, 2021 • edited Loading

Launch Checklist

karimnaaji commented Apr 16, 2021

asheemmamoowala Apr 17, 2021

Choose a reason for hiding this comment

rreusser Apr 17, 2021

Choose a reason for hiding this comment

ansis Apr 27, 2021

Choose a reason for hiding this comment

karimnaaji commented Apr 20, 2021

ansis commented Apr 21, 2021

Pitching example

rreusser commented Apr 21, 2021 • edited Loading

rreusser commented Apr 21, 2021 • edited Loading

ansis commented Apr 21, 2021

karimnaaji commented Apr 22, 2021

ansis May 6, 2021

Choose a reason for hiding this comment

ansis May 6, 2021

Choose a reason for hiding this comment

rreusser left a comment

Choose a reason for hiding this comment

rreusser May 6, 2021

Choose a reason for hiding this comment

rreusser May 6, 2021 • edited Loading

Choose a reason for hiding this comment

ansis left a comment

Choose a reason for hiding this comment

karimnaaji commented May 7, 2021

karimnaaji commented May 7, 2021

karimnaaji commented Apr 13, 2021 •

edited

Loading

rreusser commented Apr 21, 2021 •

edited

Loading

rreusser commented Apr 21, 2021 •

edited

Loading

rreusser May 6, 2021 •

edited

Loading