Improved transparency for line layers

[karimnaaji]

Changelog:

• <changelog>Improve quality of transparent line layers by removing overlapping geometry artifacts</changelog>
• <changelog>Improve performance and reduce number of draw calls for high succession of source cache/layer switches</changelog>

A draft proof of concept to improve transparency of lines. When using line layers with opacity, overlapping geometry often results in poor transparency results, where one might expect a consistent color throughout the entire line, self-intersecting geometry often results in artifacts. Refer:

• Strange rendering artefacts with the line layer #10000
• Add a per-layer opacity property #4090 (comment)
• Optimize translucent line rendering #4824

This PoC is an attempt at fixing that issue by utilizing the GPU instead of a geometric approach, as it is proven to be difficult to clean up self-intersecting polyline geometry (refer to @rreusser 's good idea from #10925).

Note that this PoC does not necessarily improve line-blur, but as noted in #4090, the most common complaints are around self-intersecting geometry that are mainly incoming from our line layers with line-opacity. Since polylines can lead to rather complex self-intersections due to narrow angles and caps.

The technique used here is to ultimately only invoke fragment operations once by stamping the stencil buffer with the polyline footprint before reverting the stamp to have the original buffer. By doing so, the self-intersecting geometry does not result in visual artifacts, since the fragment will fall-through only and exactly once.

Visualization of subsequent stencil tests:

• pass 1 draw stamp with stencil id set to invert on stencil pass and color writes on
• pass 2 revert stamp with stencil id set to invert against ~stencilId on stencil pass and color writes off

The biggest tradeoff is that lines appear more aliased, since we need to use alpha-discard to remove fragments that passed a certain opacity threshold. There's no clear workaround at the moment for that issue since we're not rendering to texture. So one open question is whether this sort of improvement would be opt-in.

Overall, aliasing is a fair tradeoff for this technique, as it appears much less noticeable to the eye than the artifact resulting from overlapping transparent geometry (refer #4090 (comment), #4090 (comment), #4090 (comment)).

The technique doesn't only apply to lines and can be extended to fill layers.

A few references and after/before from our render tests to better understand the difference:

cc @rreusser

[astojilj]

Clever. Looks great. No MSAA support in gl-native yet and it looks like this would be a problem, if enabled.

Any plans for antialiasing edges here?
GL_OES_standard_derivatives: doesn't seem like it is used currently. How do you plan to use it?

[karimnaaji]

@astojilj Yes I'm researching whether GL_OES_standard_derivatives could help on the aliasing part but no success on that yet. At the moment our option may be to expose this improvement with explicit opt-in due to that tradeoff, but if we find a way around it we could simply enable it by default when opacity != 1.0.

Something that slightly mitigates the aliasing is to reduce the alpha discard ratio until we reach a point where not too many pixels get affected by both stencil-out and alpha discard (here we can notice that it's a value somewhere around 0.7):

alpha test 1.0

alpha test 0.9

alpha test 0.7

alpha test 0.5

alpha test off

[SnailBones]

This is really neat @karimnaaji! Looks cool and seems like it would solve a lot of pain points.

            const stencilIdPass2 = ~stencilIdPass1 & 255;

I'm understanding this line correctly, the stencil is stripping out all pixels that are less than fully opaque. For more consistent appearances, would it make more sense to use a lower cutoff, maybe 50% opacity? The visual difference of diminished lines without antialising seems particularly noticeable in small shapes such as the yellow and orange lines in your second image.

[rreusser]

I think this is not useful here, but since I had it half-cobbled together and since someone mentioned my favorite extension (standard derivatives! 🎉 ), here's my standard derivative + lines antialiasing trick.

The basic idea is that you pass a varying that's -1 on one side of the line and 1 on the other. If you divide that varying by the magnitude of the gradient (length(vec2(dFdx(y), dFdy(y)))) in the fragment shader, it's magically a function you can use for antialiasing, independent of projected width! (fwidth is similar to this function and a little cheaper, but has ~40% anisotropy on the diagonals)

https://observablehq.com/d/ca328ae703c13106

The image below shows triangulated lines without MSAA, instead using alpha for antialiasing. Note that the antialiasing width is the same whether on the part of the line close to the camera or whether far away.

#extension GL_OES_standard_derivatives : enable
precision mediump float;
uniform float antialiasWidth, alpha;
varying float y;
void main () {
  float gradient = length(vec2(dFdx(y), dFdy(y)));
  gradient *= antialiasWidth;
  float antialias = min(1.0, (1.0 - abs(y)) / gradient);
  gl_FragColor = alpha * antialias * vec4(1, 0, 0, 1);
}

[karimnaaji]

This is really neat @karimnaaji! Looks cool and seems like it would solve a lot of pain points.

        const stencilIdPass2 = ~stencilIdPass1 & 255;

I'm understanding this line correctly, the stencil is stripping out all pixels that are less than fully opaque. For more consistent appearances, would it make more sense to use a lower cutoff, maybe 50% opacity? The visual difference of diminished lines without antialising seems particularly noticeable in small shapes such as the yellow and orange lines in your second image.

@SnailBones Yes I agree that it's rather noticeable. I ended up splitting the cutoff in two different passes to fix the AA. Concerning ~stencilIdPass1 & 255: we already use the stencil buffer for other purposes (mainly tile clipping to remove fragments falling outside of the tile boundaries, and fill-extrusion opacity so that only the closest pixel in terms of depth is rendered, otherwise we'd get a lot of overlapping geometry).

In that specific case, I was trying to avoid falling back on a stencil identifier that was already used for tile stencil clipping, and stencil identifiers are allocated in ascending order, which means identifiers are very local:

One thing that works is to have an identifier that falls far away from the current one, and also doesn't conflict with the tile clipping stenciling. So simply inverting it works nicely, so that stencil id 0 becomes 0xff, 1 0xfe, 2 0xfd etc.. but that's only one way of allocating identifiers, we can figure out another way if needed.

Using that, we first draw once inverting it when the stencil test passes and any fragment falls through once. Then, in a second pass revert the stamp as we need to get it back to normal, so that it doesn't conflict with any other layer, but this time against the stencil id we used for clipping inverted (~stencilId & 255).

@rreusser I love that technique but I found the derivatives difficult to leverage for the self-intersecting use case so I dropped that option for the moment and also found another way. I managed to fix the aliasing part by doing another pass and identifying which pixels contribute to AA, so the very edge of the polyline pixels are discarded in a first draw call with alpha discard threshold set to high, and then drawn once more with alpha discard threshold set to low:

stencil tests (red fail/green pass)

render

And here's the original test that @astojilj mentioned, now fixed (some tweaks can be done to adjust alpha discard threshold and prevent a few pixels from being fully rejected in the middle parts):

Refer https://103814-8629417-gh.circle-artifacts.com/0/test/integration/render-tests/index.html

But this technique doesn't work with lines having 1 pixel width, but it doesn't matter as the artifact of self-intersecting geometry isn't noticeable anyway, we can simply disable it in that case. Failed test:

The next step is to find a nice allocation strategy for the stencil identifiers, as this is not optimized since we should be able to save one draw call by simply leaving the stamp of the polyline footprint (as long as one transparent line layer stencil identifier isn't conflicting with another one). So we need to find a way to allocate stencil identifiers per line layer, while still not conflicting between different tile stencil clipping stencil ids and not reverting the stamp in an extra draw call.

[karimnaaji]

@SnailBones Concerning the next steps and to clarify a bit:

Finding a better stencil allocation strategy

At the moment the stencil id used for the stamps is directly taken from the stencil clipping mask (refer here). And, as I previously mentioned, we flip it in order to have an id associated with the line stamp that does not conflict with stencil clipping (that cuts down geometry outside of the tile bounds). This has the advantage that it's relatively simple but, since we don't create a pair line-layer/stencil-id, we have to draw it, then revert it. Otherwise different line-layers with transparent opacity will conflict between each other.

What I propose is to leave the stamp, but that means finding and creating a pair line-layer/stencil-id to skip the extra draw call here, so that different line-layer(s) do not conflict. With that approach, for example if our style contains 3 transparent line-layers, an allocation strategy could be as simple as:

• (~stencilIdPass1 & 255) - 0: line-layer 1
• (~stencilIdPass1 & 255) - 1: line-layer 2
• (~stencilIdPass1 & 255) - 2: line-layer 3

One challenge and issue with that is that we're bound by the number of potential stencil id we have (0-255) and how many are already dedicated to tile clipping, when a collision happens we could just fallback to 3 draw calls (e.g. revert the stamp) and accept it as a limitation of the approach.

Here's a more concrete case where this may happen: when drawing 100 tiles we can only draw ((~100 - 255) - 100 + 1 = 54 transparent line-layers, after we hit that hard limit we switch back to drawing the stamp and then immediately reverting it (3 draw calls instead of 2), otherwise we can just leave it in the stencil buffer (2 draw calls).

Support with terrain stencil mode

Terrain forks the stencil mode to have its own setup here. I haven't investigated in detail, but I think we should be able to do the same, we could probably create a stencilModeForTransparentLines(...) in the painter class instead of directly accessing it here.

Adding complex render test combination for different transparency

I would check whether we can use that sort of technique can be used with transparent line-gradients (https://docs.mapbox.com/mapbox-gl-js/example/line-gradient/) to achieve something like that:

It's something relatively used to create trail line for navigation trip representation.

[SnailBones]

What I propose is to leave the stamp, but that means finding and creating a pair line-layer/stencil-id to skip the extra draw call here, so that different line-layer(s) do not conflict. With that approach, for example if our style contains 3 transparent line-layers, an allocation strategy could be as simple as:

• (~stencilIdPass1 & 255) - 0: line-layer 1
• (~stencilIdPass1 & 255) - 1: line-layer 2
• (~stencilIdPass1 & 255) - 2: line-layer 3

In the example image you shared, six tiles are visible onscreen with ids 0 through 5.¹ If I understand the allocation strategy correctly, drawing a line on these tiles will use ids 255 down to 250. So then the above approach would create conflicts between the different line layers, and would need to be something like:

• (~stencilIdPass1 & 255) - 0: line-layer 1
• (~stencilIdPass1 & 255) - 6: line-layer 2
• (~stencilIdPass1 & 255) - 12: line-layer 3

Or in the general case:

• (~stencilIdPass1 & 255) - n * tileCount: line-layer n

Here's an example of what IDs on the stencil buffer might look like with six tiles and two transparent line layers:

Since the stencil buffer is now shared between tile clipping and line layers, any approach that leaves a change (i.e. any approach without the extra draw call) would require a more complex approach to tile clipping IDs. For instance with the above approach to correctly clip tile t all painting would now need to check for stencil IDs n, 255-t, 255-tile_count-t, 255-tile_count*2-t and so on. It seems to me that the easiest way to implement something like this would be using the stencil buffer's bitmasking feature. (My WebGL knowledge is limited so there may be other options I'm not considering.)

By effectively giving every transparent line layer its own one-bit mask, bitmasking would also solve the problem of intersections. Here's the above example with this allocation strategy:

In this case, late bits are used for tile clipping, and each of the first two bits (128 and 64) is dedicated to a line layer.
Most tile rendering would ignore the first two bits by using the mask 00111111 (63 or 0x3F). To get the mask to paint a tile layer correctly, we OR the tile mask above with the tile layer's dedicated bit: in this case the purple layer would use the mask 10111111 (191 or 0xBF).

The stencil buffer has only has 8 bits, so we'll still run out of space pretty quickly. Where n is the number of transparent line layers, we get a maximum of 2^(8-n) tiles onscreen: we could combine 4 line layers with 16 tiles or 5 line layers with 8 tiles. Additional line layers could use the extra draw call to revert the stencil buffer. I think it should be possible to allocate these bits dynamically as needed but I might be wrong. If not, we need to make sure that we're saving enough space for the maximum possible number of visible tiles.

Footnotes

1. I'm also curious to how you generated that image, I presume it's just a mockup and it's just wishful thinking that we have a tstencil buffer visualizing debug mode? ↩

[mourner]

The code looks great, such a thoughtful PR! 🙏 Did you try running the benchmark suite on the approach? I'm curious whether it impacts rendering performance and how much if so.

[karimnaaji]

Did you try running the benchmark suite on the approach? I'm curious whether it impacts rendering performance and how much if so.

Thanks for giving a look @mourner , I'll do benchmarking next. I expect the benchmark suite to run the same, as default styles don't overly make use of transparency, so this will need a bit more fine-grained profiling to be aware of the change. But as we may draw lines twice, we can expect render times to be higher for that case. Two optimizations have been added in this PR to mitigate that:

• Rendering in 2 draw calls instead of 3 (as initially proposed) to revert the stamp we make in the stencil: This leverages the fact that rendering tile clipping should always be lesser than rendering of lines, as they have more complex geometry/shader. So instead of reverting the lines by drawing them again after the two first render pass, we revert the stamp made in the stencil buffer by drawing the quad covering the entire tile. This is more fragment processing, but stencil clipping drawing is faster nonetheless.
• Reuse stencil tile clipping of tiles by not drawing equivalent tile set, even when the sources differ, as mentioned in Optimize stencil clipping masks render for high source cache/layer switches #11197.

[karimnaaji]

@mourner Fine-grained profiling (captured using gpu-time-layer) on this location with a style having using 4 transparent layers (knaaji/ckolnqm010uk717qlv51vgb1c), shows a diff of 0.14041415460122603ms (old, averaged over 1000+ frames) vs 0.2297283694857026ms (new, averaged over 1000+ frames) of GPU time difference for the render time of these layers. So transparent lines are ~1.6x slower to render, but is expected due to the extra draw call that this technique uses.

Higher-level metrics and profiling do not show a difference on our benchmark suite:

I added a few render tests to finalize the PR. I think it's ready if we're ok with the tradeoff of render time mentioned above for improved quality.

[mourner]

So transparent lines are ~1.6x slower to render, but is expected due to the extra draw call that this technique uses.

How common are translucent lines on typical styles like streets? 1.6x is a serious slowdown so let's make sure is doesn't affect most users. If it does, did we consider making an opt-in style-spec switch for this behavior?

[karimnaaji]

How common are translucent lines on typical styles like streets?

Verified and it's not common. I observed at most one layer, usually admin-0-boundary-bg used as a double layered lines. As for numbers for these styles, the impact is very minimal. All the numbers below are GPU time in ms using gpu-time-layer averaged over 1000+ frames as per previous comment:

	karim/stencil-lines-poc	main	layers
streets-v11	0.07726632657926079	0.04087094759358228	admin-0-boundary-bg
satellite-streets-v11	0.04430581684981622	0.05473528179551118	admin-0-boundary-bg
light-v10			N/A
dark-v10			N/A
outdoors-v11			N/A
navigation-day-v1	0.04279654756380543	0.03417142857142915	admin-0-boundary-bg
navigation-night-v1	0.032405662145499524	0.03051212030075214	admin-0-boundary-bg

In these styles transparent-lines are usually of more minimal use, or use lighter width to cover these artifacts, as they quickly show up otherwise.

I've considered opt-in but it might not work well for these type of improvements, as it reduces discoverability and I really think we should try to give the best expected visual quality out of the box. But I'm open to more thoughts about it.

Reuse stencil tile clipping of tiles by not drawing equivalent tile set, even when the sources differ, as mentioned in Optimize stencil clipping masks render for high source cache/layer switches #11197.

I'd also like to reiterate that point. Which is an optimization added in this PR that reduces the number of some draw calls by almost half on navigation-night/day-v1 (or any style that have high source cache/layer switches):

• drawElements 444 calls reduced to 276
• uniformMatrix4fv 408 calls reduced to 241
• stencilFunc 346 calls reduced to 178

I've extended the profiling to the GPU for the styles navigation-day-v1 and navigation-night-v1 (full GPU frame time):

	karim/stencil-lines-poc	main
navigation-day-v1	1.5023987073891578	2.0252601594571593
navigation-night-v1	1.0294184446546841	2.2894521608579144

Also refer to #11197, which is a cost that is now entirely removed as it was redundant:

Profiling shows that about 6ms out of 11.2ms of GPU time (6049.92us out 11226.208us) were spent on DrawIndexed(6) which is the tile clipping render

[mourner]

Great!

[Amber-372]

Can this optimization be merged into version 1.x ?

[mourner]

@Amber-372 no.

[c-harding]

@mourner @karimnaaji is it possible to turn this “improvement” off? I had been using this for line-based heatmaps (brighter = more overlapping tracks), which this change breaks. Or do I need a couple of thousand layers to represent a couple of thousand tracks? This will still break tracks with multiple laps, which would only contribute one “level” of brightness to the map.

Before (Mapbox 2.6):

After (Mapbox 2.12):

[karimnaaji]

@mourner @karimnaaji is it possible to turn this “improvement” off? I had been using this for line-based heatmaps (brighter = more overlapping tracks), which this change breaks. Or do I need a couple of thousand layers to represent a couple of thousand tracks?

@c-harding Yes it's possible.

For context, consistent line opacity throughout the line is what most users expect by default for the majority of our use cases. It has been reported as a rendering artifact for a while before we introduced this change. Leveraging the opacity for line heatmap is less frequent, and something we could improve with a dedicated native layer to facilitate its use (which would allow to adjust the range of the possible values, how the opacity gets composited, and what color gradient to apply for that layer etc..).

With that being said, we only enabled this change when used on line-opacity , not on the alpha channel of the line-color. You can revert to the old behavior by transitioning your value from line-opacity to the alpha channel of the line-color: