Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to set Output Vertex Queue size within pipeline API #3071

Open
ashley-taylor opened this issue Oct 20, 2021 · 0 comments
Open

Ability to set Output Vertex Queue size within pipeline API #3071

ashley-taylor opened this issue Oct 20, 2021 · 0 comments

Comments

@ashley-taylor
Copy link

Hello,

We have a keyed Aggregation that computes an array of substantial objects.
The output of these aggregation goes straight into a flat map to explode this array.
It would be awesome if there were a way to return a Traverser from an aggregation.

Each object in the array can be in the 100's of MB, and the array can contain 100's of elements.

With the default queue settings, the ram requirements to queue this with the default settings are large.

We used the toDag method to convert the pipeline then manually set the outbound queue sizes.
This approach does works. It is a bit inconvenient and does rely on the deprecated toDag method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants