Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(bigquery/storage/managedwriter): introduce send optimizers #7323

Merged
merged 4 commits into from
Jan 31, 2023
Merged

refactor(bigquery/storage/managedwriter): introduce send optimizers #7323

merged 4 commits into from
Jan 31, 2023

Conversation

shollyman
Copy link
Contributor

This PR introduces a new optimizer abstraction for connection objects, but doesn't wire it in (happening in another open PR).

The purpose of the optimizers is to leverage awareness of previous requests to reduce transferred bytes. We provide three optimizers, based on the expected usage:

  • passthroughOptimizer - used for testing (no-op optimizer)
  • simplexOptimizer - used for connections where only a single stream will be sending data (e.g. explicit streams)
  • multiplexOptimizer - used for connections shared by multiple writers

Towards: #7103

@shollyman
refactor(bigquery/storage/managedwriter): introduce send optimizers
9fcf41f 
This PR introduces a new optimizer abstraction for connection objects,
but doesn't wire it in.  The purpose of the optimizers is to leverage
awareness of previous requests to reduce transferred bytes.
@shollyman shollyman requested review from alvarowolfx, yirutang and a team January 30, 2023 06:42
@shollyman shollyman requested a review from a team as a code owner January 30, 2023 06:42
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the BigQuery API. labels Jan 30, 2023
alvarowolfx
alvarowolfx approved these changes Jan 30, 2023
View reviewed changes
Copy link
Contributor

@alvarowolfx alvarowolfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shollyman shollyman merged commit c37f9ae into googleapis:main Jan 31, 2023
@shollyman shollyman deleted the send-optimizer branch January 31, 2023 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alvarowolfx alvarowolfx alvarowolfx approved these changes

@yirutang yirutang Awaiting requested review from yirutang

Assignees
No one assigned
Labels
api: bigquery Issues related to the BigQuery API. size: l Pull request size is large.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@shollyman @alvarowolfx