New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added rate-limiter configs in the table level #563
base: main
Are you sure you want to change the base?
Conversation
@ParthaI could you please resolve the merge conflicts in this PR? Additionally could you please add the logs wherein you can display the results with and without the rate-limiters and how it affects the overall working of the plugin? |
BucketSize: 100, | ||
MaxConcurrency: 50, | ||
Scope: []string{"connection", "service", "action"}, | ||
Where: "service = 'bigquery' and (action = 'datasets.list' or action = 'datasets.get')", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ParthaI should the Where
section be as follows, since you mentioned in the comment that the rate limiter would be beneficial for the LIST operations?
Where: "service = 'bigquery' and action = 'datasets.list'",
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@misraved, as per the comment we are still getting the rate limiter error if we run the query select * gcp_bigquery_dataset
. If we are only configuring the limiter for the LIST API call. Because we are also making the GET API call as it is being used as a hydrated API call.
If we encounter a rate-limit error during a LIST API call and simultaneously make a GET API call for the same service, we will likely face the same rate limit error. Therefore, I have configured both the LIST and GET calls within the where clause to address this issue.
Integration test logs
Logs
Example query results
Results