Fix precision errors in pack/unpack depth

[jtorresfabra]

Current code is introducing precision errors in the pack/unpack depth functions by not using the correct range of values. When the packed_depth is written to R8G8B8A8 texture the maximum value it can store is 255 per channel, not 256. We were introducing errors in the third decimal in a range from 0..1. Seems this code was also wrong in three.js and other libraries, hence the confusion. Coded a godbolt to prove the issue (https://godbolt.org/z/8MrrhPh11).

Link to the original post: https://aras-p.info/blog/2009/07/30/encoding-floats-to-rgba-the-final/

This fix also helps in gl-native terrain occlusion and others.

Tested manually that the problem addresed in this commit is still fixed.

cc: @mapbox/gl-native

Launch Checklist

• briefly describe the changes in this PR
• include before/after visuals or gifs if this PR includes visual changes
• manually test the debug page
• tagged @mapbox/map-design-team @mapbox/static-apis if this PR includes style spec API or visual changes
• tagged @mapbox/gl-native if this PR includes shader changes or needs a native port
• apply changelog label ('bug', 'feature', 'docs', etc) or use the label 'skip changelog'
• add an entry inside this element for inclusion in the mapbox-gl-js changelog: <changelog>Fix precision errors in depth pack/unpack</changelog>

[mpulkki-mapbox]

@jtorresfabra this is great nice find! I've been investigating incoherent behavior of the depth texture in the past but never got this far. I created few charts to better understand how the encoding function works on different input. Values in the charts represents relative error of the unpacked value compared to the actual value.

Broken

Fixed

Fixed version of the encoding provides much better precision where the maximum deviation is ~3% in my tests. For example value close to 0.132 produces some difference. Overall the error is constrained and could be used to deduce maximum required bias value when comparing depth values.