Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logBase implementation is insufficiently precise for base 10 #1048

Open
rriegs opened this issue Oct 9, 2019 · 0 comments · May be fixed by #1054
Open

logBase implementation is insufficiently precise for base 10 #1048

rriegs opened this issue Oct 9, 2019 · 0 comments · May be fixed by #1054
Labels

Comments

@rriegs
Copy link

rriegs commented Oct 9, 2019

, test "logBase 10 100" <| \() -> Expect.equal 2 (logBase 10 100)

As above, a unit test exists checking logBase 10 100, however if this had checked logBase 10 1000, it would have caught the fact that logBase 10 x implemented as log x / log 10 (using JavaScript's natural logarithm) can yield results slightly less than an integer even for small power of 10 inputs. This unfortunately complicates a common use case for logBase 10: finding it's floor in order to determine a exponent when formatting a number in scientific notation.

JavaScript's log10 is provided separately specifically to address this imprecision, and indeed it returns an integer for all powers of 10 ranging from -311 to 308 (i.e., all those representable as normal IEEE 754 binary64 floats and even a few of the subnormals). Accordingly, I respectfully recommend either adding a corresponding log10 function in Elm or a special case for logBase 10 that instead calls JavaScript's log10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 participants