Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Smarter bulk & unordered scans #43

Open
ashvardanian opened this issue Aug 28, 2022 · 2 comments
Open

Smarter bulk & unordered scans #43

ashvardanian opened this issue Aug 28, 2022 · 2 comments

Comments

@ashvardanian
Copy link
Contributor

Currently ukv_scan is only working for fully consistent sorted exported of keys from collections.
With the bulk flag we allow prioritizing throughput over consistency, but a point can be made, that ML-like pipelines don’t need any dependency in operations whatsoever. Instead they may use scans to uniformly random-sample entries, which would in turn require a full scan of keys. If the user leaves start_key unset, we can perform the bulk sampling behind the curtains ourselves.
It will make the interface more ugly by making a function dual-use, but will keep the interface short. Worth considering.

@ashvardanian ashvardanian self-assigned this Aug 28, 2022
@ashvardanian
Copy link
Contributor Author

Those changes should preceed #17 to have a finalized scan interface.

ashvardanian added a commit that referenced this issue Sep 5, 2022
Fix: Python build with new scans #43
Fix: retrieving the gist of document fields.
@ashvardanian ashvardanian added this to the 0.4.0 milestone Oct 3, 2022
@ashvardanian ashvardanian changed the title Smarter scans Smarter bulk & unordered scans Oct 18, 2022
@ashvardanian
Copy link
Contributor Author

If the bulk flag is provided, we can treat the passed keys as not start keys but instead the last keys in the previous batch.

DarvinHarutyunyan pushed a commit that referenced this issue Dec 9, 2022
Fix: Python build with new scans #43
Fix: retrieving the gist of document fields.
@ashvardanian ashvardanian modified the milestones: 0.4.0, 0.5.0 Dec 10, 2022
@ashvardanian ashvardanian removed their assignment Jan 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: No status
Development

No branches or pull requests

1 participant