Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: make get_results lazy #1632

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

sk1p
Copy link
Member

@sk1p sk1p commented Apr 25, 2024

This is still work-in-progress, and slated for the 0.15 release. It's an absolutely necessary performance fix in case get_results does any significant work (like CoM with regression=1), and is especially useful in combination with rate limiting visualization results, also pushing us one step further towards not having to manually touch the partition sizes together with LiberTEM-live (we can just make them smaller by default!).

Main TODOs:

  • How to make this thread-safe - we need to prevent the delayed get_results and the merge function from running concurrently on the same data / buffers, basically.
  • Needs documentation on how to safely access the results of run_udf_iter
  • This slightly changes the observed behavior, as seen in the failing test cases. I think this should be OK and the tests and the docs need to be updated, but some input would be welcome! This is mostly an issue if get_results has observable side-effects

Fixes #1596

Contributor Checklist:

Reviewer Checklist:

  • /azp run libertem.libertem-data passed
  • No import of GPL code from MIT code

@sk1p sk1p added enhancement New feature or request UDFs Anything related to the user-defined functions interface labels Apr 25, 2024
@sk1p sk1p added this to the 0.15 milestone Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request UDFs Anything related to the user-defined functions interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow to call get_results lazily
1 participant