Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LazyFrame.filter fails to parse non-Sequence iterables #16253

Closed
2 tasks done
dangotbanned opened this issue May 15, 2024 · 0 comments · Fixed by #16254
Closed
2 tasks done

LazyFrame.filter fails to parse non-Sequence iterables #16253

dangotbanned opened this issue May 15, 2024 · 0 comments · Fixed by #16254
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@dangotbanned
Copy link
Contributor

dangotbanned commented May 15, 2024 •
edited

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.DataFrame({"foo": [1, 2, 3], "bar": [6, 7, 8], "ham": ["a", "b", "c"]})
predicate = [pl.lit(True)]
output_1 = df.filter(predicate)
output_2 = df.filter(iter(predicate))

Log output

No response

Issue description

Discovered this when passing in a generator, which should be acceptable as Iterable[IntoExprColumn].
The same input is allowed for select, with_columns, etc as they first consume the generator.

Proposing a fix in #16254

Expected behavior

output_1 and output_2 should be identical, based on the annotation for LazyFrame.filter.

>>> output_1
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ 6   ┆ a   │
│ 2   ┆ 7   ┆ b   │
│ 3   ┆ 8   ┆ c   │
└─────┴─────┴─────┘

>>> output_2
File ...\Lib\site-packages\polars\lazyframe\frame.py:2815, in LazyFrame.filter(self, *predicates, **constraints)
   2809     err = (
   2810         f"Series(…, dtype={p.dtype})"
   2811         if isinstance(p, pl.Series)
   2812         else repr(p)
   2813     )
   2814     msg = f"invalid predicate for `filter`: {err}"
-> 2815     raise TypeError(msg)
   2816 else:
   2817     all_predicates.extend(
   2818         wrap_expr(x) for x in parse_as_list_of_expressions(p)
   2819     )
TypeError: invalid predicate for `filter`: <list_iterator object at 0x0000015DF007BCD0>

Installed versions

--------Version info---------
Polars:               0.20.26
Index type:           UInt32
Platform:             Windows-10-10.0.19045-SP0
Python:               3.12.3 | packaged by conda-forge | (main, Apr 15 2024, 18:20:11) [MSC v.1938 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               2.2.2
pyarrow:              16.0.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             0.8.1
xlsxwriter:           3.1.9
@dangotbanned dangotbanned added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels May 15, 2024
@dangotbanned dangotbanned mentioned this issue May 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(python): Handle non-Sequence iterables in filter dangotbanned/polars
1 participant
@dangotbanned