-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support numpy types #31
Comments
+1 |
Actually, |
Can't deny the same regarding the #24 but I guess it can be good if we have all of the types in one package! |
Maybe the extra |
I agree with both options - either displaying a message that prompts the user to install Numpy, or proceeding with the extra requirements since it functions properly in our current scenario, this will allow for the possibility of adding any big package as extra |
Hey folks, Just so I understand a little bit better our objective with the Numpy(value=np.float64(12)) Do we want to have some conversions as well ? because my first idea was to validate using The reason why I am asking this is the following: if we want to do those conversions we would probably need to make some decisions (e.g., we have If my reasoning is not fair, just let me know 😄 |
@kroncatti Pydantic has two modes of parsing/validation: strict and lax. In lax mode it already coerces types in many cases such as this: class Model(BaseModel):
foo: int
m = Model(foo='12')
print(m)
# foo=12
print(type(m.foo))
# <class 'int'> So, it seems natural for Pydantic to coerce values to numpy types on validation. |
Thanks @lig, Just to check if I properly understood. If we set: class Model(BaseModel):
foo: Numpy
m = Model(foo='12')
print(m)
# foo=12
print(type(m.foo))
# <class 'numpy.int64'> This should be the outcome ? Shouldn't the user have to specify what numpy type we are going to coerce ? such as int64, float64, etc. |
I think the idea here is to support the following types: https://numpy.org/doc/stable/user/basics.types.html#array-types-and-conversions-between-types We should create the following, and all the analogous:
I guess we also want |
That makes sense. so we are basically creating one extra type for each one of those types instead of having a generic type for all of them. Cool! |
Hey guys,
|
Hello again, Maybe it can be added to the main branch as a start for support of numpy types. |
One of the issues to deal with is that JSON cannot natively represent all the dtype's for numpy arrays without extra context. So we would have to choose a reasonable schema as a default for the serializing/deserializing to be isomorphic. Namely, we need to have an opinionated default for representing arrays of complex numbers. There are of course many valid options so having it be easily overwritable would also be useful. That being said, the way we have been solving is as well as the ASE project has been with {'__ndarray__': (
obj.shape,
str(obj.dtype),
flatobj.tolist()
)} The serializing / deserializing logic is in the file I linked. |
I suggest you to check out my project |
I'd personally prefer that pydantic/pydantic-extra-types natively support numpy types.
My use case is that I want to define a metadata standard for ml models that take large and complex arrays. the metadata just needs to record the numpy type, order of labeled dimensions, and the shape. Saving out the whole array to load into the Numpy Model isn't preferrable, it would be uneccessary storage and susceptible to path errors |
The co-author of
If you only want validation, and dimension enforcement, just use
@rbavery your statement is false, interaction with numpy files or saving/loading is an optional quality of life feature, and is only offered with Please be careful when you make these claims, I'd rather have you ask the question than claiming if you are uncertain. Update Refactoring of pydantic-numpy.typing: We're giving the submodule a minor overhaul. The key change is the transition to automatically generated code. This shift is essential since dynamically generated typing hints aren't compatible with static type checkers like MyPy and PyRight. The refactor is compatible with static type checkers, and the code generator's script is available in the repository for reference. Introducing NumpyModel for File IO: We've added a NumpyModel that supports integrated file IO operations. This should streamline processes that involve NumPy data handling. Comprehensive Testing for Coverage: We've also put a significant effort into extensive testing to ensure robust coverage and reliability. Regarding the integration of Complexity Management: Merging Community Feedback: I'm aware of the requests to incorporate NumPy types directly into this repository. While I understand the perspective, I believe maintaining separation is in our best interest for streamlined development and maintenance. Documentation Update: To make |
Changes made because of an alarming interpretation of the package on pydantic/pydantic-extra-types#31
Changes made because of an alarming interpretation of the package on pydantic/pydantic-extra-types#31
woops sorry @caniko, my bad. I read the readme incorrectly. Thanks for correcting. |
The idea here is to support the numpy types mentioned on https://numpy.org/doc/stable/reference/arrays.scalars.html.
The text was updated successfully, but these errors were encountered: