Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gen.realFrac_ can produce values outside of the bounds of the Range (and might miss values) #441

Open
jcarr-awake opened this issue Dec 8, 2021 · 0 comments

Comments

@jcarr-awake
Copy link

jcarr-awake commented Dec 8, 2021

The realFrac_ generator uses the method realToFrac and a double scalar in order to smoothly interpolate between the two endpoints of the range. This causes types to round trip through Double, and may cause them to lose precision. While this is perhaps undesirable on the middle of the range by missing values, it can be particular problematic on the edges. There is no guarantee for RealFrac that this round-trip should preserve the value, and several standard library types such as Fixed do not preserve it.

On the edges, it can cause values to be generated which are below the lower bounds of the range, or above the upper bounds.

To fix this issue, the result value should be clamped to the range, and the function should include warning documentation about possible loss of precision on the inside of the range for high-precision types.

An example value which changes when converted to Double and back (generated randomly by Hedgehog)
357559679.487764797 :: Nano
becomes
357559679.487764775

So if your lower bound was 357559679.487764796 then this could have been incorrectly generated by a realFrac_ generator on Nano

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant