Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decimal comma in Float32 in JSON #201

Open
jerro-org opened this issue Feb 25, 2020 · 9 comments · May be fixed by #528
Open

Decimal comma in Float32 in JSON #201

jerro-org opened this issue Feb 25, 2020 · 9 comments · May be fixed by #528
Labels
Bug Something isn't working

Comments

@jerro-org
Copy link

If I set the locale of my Ubuntu System to "de_DE.UTF-8" oat++ serializes float numbers with a comma as decimal separator instead of a dot.
When executing a GET using the Swagger UI I get:

can't parse JSON. Raw result:
{"x":2,000000,"y":0,000000,"z":200,000000}

Is this legal JSON or a bug in oat++ or can this be configured somehow?

@lganzzzo lganzzzo added the Bug Something isn't working label Feb 25, 2020
@lganzzzo
Copy link
Member

Hello @jerro1971 ,

Thanks for reporting this issue.
It's a bug 💯 !

We'll have to fix it.

Regards,
Leonid

@bamkrs
Copy link
Member

bamkrs commented Apr 2, 2020

My 2c to that.
It all boils down to using snprintf in

v_buff_size float32ToCharSequence(v_float32 value, p_char8 data, v_buff_size n) {

An easy - but not elegant - workaround would be to set the locale for numerics to en_US or similar if they are not json-compatible.
So we could check in oatpp::base::environment::init() if the locale is compatible. If not, we could set a flag that it should be changed when formatting. But we need to reset the locale to the original! We don't want to screw around with the locales for the entire program!

#include <locale>
// [...]
v_buff_size float32ToCharSequence(v_float32 value, p_char8 data, v_buff_size n) {
    char* old_locale;
    if(oatpp::base::environment::switchToJSONCompatibleLocale) {
        old_locale = strdup(setlocale(LC_NUMERIC, nullptr);
        setlocale(LC_NUMERIC, "en_US"); // or plain "C" locale for portability;
    }
    auto rc = snprintf((char*)data, n, "%f", value);
    if(oatpp::base::environment::switchToJSONCompatibleLocale) {
        setlocale(LC_NUMERIC, old_locale);
        free(old_locale);
    }
    return rc;
  }

However, this is not very elegant and race-conditions could occur!

A c++ way would be to use a stringstream to do that, but I don't know about its performance:

#include <iostream>
#include <locale>
 
class force_dot_decimal : : public std::numpunct<char> {
  protected:
    virtual char do_decimal_point() const
    {
        return '.';
    }
};
 
v_buff_size float32ToCharSequence(v_float32 value, p_char8 data, v_buff_size n) {
{
    std::stringstream ss;
    /* 
     * This way we use the standard locale from std::locale() but force it to use a point as decimal point
     * and we don't need to force the user to install our predefined en_US locale
     */ 
    ss.imbue(std::locale(std::locale(), new force_dot_decimal));
    ss << value;
    /* [...] copying ss to data */
}

At least for BSD, OSX and Windows we could use snprintf_l with the en_USlocale... But it would not work for linux (shame!).

One (even more ugly hack) would be to brute-force a replacement...

v_buff_size float32ToCharSequence(v_float32 value, p_char8 data, v_buff_size n) {
    auto rc = snprintf((char*)data, n, "%f", value);
    p_char8 ptr = data;
    for(v_buff_size c = 0; c < rc; ++c){ // or use a stdlib function to search for ','
        if(*ptr == ',') {
           *ptr = '.';
        }
    } 
    return rc;
  }

@lganzzzo
Copy link
Member

lganzzzo commented Apr 2, 2020

Hey @bhorn ,

Yes, really, the fix of this issue is frustrated by existing options.

I am more down to the last proposed variant with comma replacement.
And actually implementing float-to-string method from scratch, in this case, looks even more appealing to me.

Also, I think that the fix to this issue should be affecting JSON serializer/deserializer functionality only.
Meaning that the egular floatToStr methods from conversion utils should still be local sensitive.

@bamkrs
Copy link
Member

bamkrs commented Apr 2, 2020

Yes, really, the fix of this issue is frustrated by existing options.

True that, thats literally the first issue that made me say *oof*

And actually implementing float-to-string method from scratch, in this case, looks even more appealing to me.

I took a peek at nlohmann's json-lib. This is one hell of a task.

Also, I think that the fix to this issue should be affecting JSON serializer/deserializer functionality only.
Meaning that the egular floatToStr methods from conversion utils should still be local sensitive.

Agreed

I am more down to the last proposed variant with comma replacement.

Maybe we should implement this for now as hot-fix and come up with something more useful later?

@lganzzzo
Copy link
Member

lganzzzo commented Apr 2, 2020

And actually implementing float-to-string method from scratch, in this case, looks even more appealing to me.

I took a peek at nlohmann's json-lib. This is one hell of a task.

Maybe we should implement this for now as hot-fix and come up with something more useful later?

I Agree

@jerro-org
Copy link
Author

How about:
https://stackoverflow.com/questions/1333451/locale-independent-atof
This is for the other way round. But imbue() should also work for an output stream...

@lganzzzo
Copy link
Member

lganzzzo commented Apr 2, 2020

Yes, this was also considered in the @bhorn 's post.
It will work, but the downside of this method is the performance. - (Actually, I haven't test it, but I believe so) because we'll need to copy data from stream to the buffer provided.

@LokleLama
Copy link

I have a solution which works good for my project. I attached the patches for version 1.2.0...
oatpp.zip

@bamkrs
Copy link
Member

bamkrs commented Dec 14, 2020

Hey @LokleLama Could you create a fork and a PR for that so can review the changes in a more streamlined process?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants