You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Hi, we have been using GE before in our team but we are migrating most of our pandas processing pipelines to polars. However it seems that GE does not have any support for polars and transforming the polars dataset to pandas just to run the GE suite seems extremely subotimal.
Describe the solution you'd like
We would like to get polars support in GE.
Describe alternatives you've considered
We have considered to use pandera, which is going to offer full polars support very soon:
Additional context
We are in the situation were our data is not big enough to use spark. Therefore GE with pyspark or pydequee have been discarded for data validation.
The text was updated successfully, but these errors were encountered:
Jumping on the bandwagon here.
We are also exploring migrating from pandas to polars in our data processing pipelines.
Pandera is working on this and it appears support for polars is on its way there.
Polars is blazingly fast compared to pandas and this would be incredibly helpful if GX added support for a polars execution engine.
Is your feature request related to a problem? Please describe.
Hi, we have been using GE before in our team but we are migrating most of our pandas processing pipelines to polars. However it seems that GE does not have any support for polars and transforming the polars dataset to pandas just to run the GE suite seems extremely subotimal.
Describe the solution you'd like
We would like to get polars support in GE.
Describe alternatives you've considered
We have considered to use pandera, which is going to offer full polars support very soon:
unionai-oss/pandera#1064
https://pandera--1373.org.readthedocs.build/en/1373/polars.html#polars
Additional context
We are in the situation were our data is not big enough to use spark. Therefore GE with pyspark or pydequee have been discarded for data validation.
The text was updated successfully, but these errors were encountered: