Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added rate-limiter configs in the table level #563

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

ParthaI
Copy link
Contributor

@ParthaI ParthaI commented Apr 9, 2024

Integration test logs

Logs
Add passing integration test logs here

Example query results

Results
Add example SQL query results here (please include the input queries as well)

@ParthaI ParthaI self-assigned this Apr 9, 2024
@ParthaI ParthaI marked this pull request as draft April 9, 2024 16:47
@ParthaI ParthaI requested a review from misraved April 18, 2024 04:35
@ParthaI ParthaI marked this pull request as ready for review April 18, 2024 04:35
@misraved
Copy link
Contributor

@ParthaI could you please resolve the merge conflicts in this PR?

Additionally could you please add the logs wherein you can display the results with and without the rate-limiters and how it affects the overall working of the plugin?

BucketSize: 100,
MaxConcurrency: 50,
Scope: []string{"connection", "service", "action"},
Where: "service = 'bigquery' and (action = 'datasets.list' or action = 'datasets.get')",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ParthaI should the Where section be as follows, since you mentioned in the comment that the rate limiter would be beneficial for the LIST operations?

Where:          "service = 'bigquery' and action = 'datasets.list'",

Copy link
Contributor Author

@ParthaI ParthaI Apr 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@misraved, as per the comment we are still getting the rate limiter error if we run the query select * gcp_bigquery_dataset. If we are only configuring the limiter for the LIST API call. Because we are also making the GET API call as it is being used as a hydrated API call.

If we encounter a rate-limit error during a LIST API call and simultaneously make a GET API call for the same service, we will likely face the same rate limit error. Therefore, I have configured both the LIST and GET calls within the where clause to address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants