Enable roundtrip pydantic->json->pydantic for float +- infinity and nan

[hassec]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

I've seen #1885 and other np.inf, float(inf), float(nan) related issues but it's not quite clear if the below is really an intended behavior.

A pydantic model will by default allow nan and inf for float values, thus I would like to be able to serialize these models to and then also from json. But that's currently not possible because those special float values are turned into 'null' in json and thus raise an error when trying to turn json back into a pydantic model.

If this is indeed intended behavior, would an additional flag to model_dump_json be possible that would enable a similar behavior to python's json module?

Example Code

from pydantic import BaseModel
import json

class A(BaseModel):
   a : float
 
a = A(a='inf')
type(a.a), a.a  # (float, inf)
a.model_dump_json()  # '{"a":null}'
A.parse_raw(a.model_dump_json())  # raises error

json.dumps(a.model_dump()) # '{"a": Infinity}'
json.loads(json.dumps(a.model_dump())) #  {'a': inf}

Python, Pydantic & OS Version

pydantic version: 2.1.1
        pydantic-core version: 2.4.0
          pydantic-core build: profile=release pgo=true mimalloc=true
                 install path: ..../lib/python3.10/site-packages/pydantic
               python version: 3.10.12 (main, Jul 19 2023, 11:53:29) [GCC 12.2.0]
                     platform: Linux-6.2.0-26-generic-x86_64-with-glibc2.37
     optional deps. installed: ['typing-extensions']

Selected Assignee: @Kludex

[Kludex]

Yeah, it's an issue. 👍

cc @davidhewitt

[davidhewitt]

@samuelcolvin it looks like this will take some effort because serde_json doesn't accept Infinity or NaN as inputs. However I think it's a definite regression from V1.

[mhearne-usgs]

Is there a workaround for this? The only thing I can think of is in my model changing this:

value: float

to

value: Optional[float] = None

I'd prefer to have some way to read/write NaN values in the JSON...

[hvmptydvmpty]

What is the plan though, just wait if/when serde includes such support? Looks like it's been four major releases of V2 already with this regression and it's significant. One example are non-nullable columns, it's much more efficient to keep NaN in arrow non-nullable float or decimal columns than use null (and nullable columns) so keeping data-heavy APIs (that can't get off JSON for various reasons) working correctly while working around this is a major pain.

[ghost]

Is there any possible workaround for this using custom serializers to serialize nan and inf to strings?

[hassec]

@davidhewitt I saw that you fixed pydantic-core in pydantic/pydantic-core#1062, thank you! 🙏

Is this already useable from pydantic 2.5.0, or will we have to wait for an update to ConfigDict to enable this mode?

[samuelcolvin]

It should work in pydantic.

[hassec]

@samuelcolvin, not via model_config though, or am I doing something wrong?

[ghost]

I can also confirm that ConfigDict does not yet seem to support ser_json_inf_nan. The above approach therefore does not work in pydantic 2.5.1.

[davidhewitt]

Sorry yes this got closed before I managed to add the config setting to pydantic, will push a PR shortly...