Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bigquery Go client: Retry 503s on Throttle error #981

Closed
ruiyang2015 opened this issue Apr 25, 2018 · 6 comments
Closed

Bigquery Go client: Retry 503s on Throttle error #981

ruiyang2015 opened this issue Apr 25, 2018 · 6 comments
Assignees
@jba
Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@ruiyang2015
Copy link

We use Bq go client to fetch table data (using the tabe data list),
but from time to time we get a hard failure of We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
It seems Bq Client is doing the Retry for 502 and other temporary errors, but not this one, will it be possible to retry on this error in the Bigquery Golang client?
@jeanbza jeanbza added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Apr 25, 2018
@jba
Copy link
Contributor

jba commented Apr 26, 2018

Can you give us more information about the error, like the status code?

@jba jba self-assigned this Apr 26, 2018
@ruiyang2015
Copy link
Author

ruiyang2015 commented Apr 26, 2018 •
edited
Loading

The status code is 503, and the detailed message from Golang BigQuery API is like this:
Bigquery stream error while reading one row: googleapi: got HTTP response code 503 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>,None)

@ruiyang2015
Copy link
Author

We also added log at line: https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/bigquery/bigquery.go#L160, and it seems this 503 error is not treated as retryable error.
We are getting this error inside the RowIterator, and it seems there is no way for us to add our own retry logic, and when this error happens, we have to start from scratch again, our tables are usually huge, with rows in the 100M level, restart from the beginning is too costly for us.

@ruiyang2015
Copy link
Author

FWIW, we got 403 error sometimes as well, like this one, and it does not get retried too:

Bigquery stream error while reading one row: googleapi: Error 403: retry failed with context deadline exceeded; last error: Exceeded rate limits: Your 878926448773 exceeded quota for tabledata.list bytes per second per project. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors\\nMore details:\\nReason: rateLimitExceeded, Message: Exceeded rate limits: Your 878926448773 exceeded quota for tabledata.list bytes per second per project. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

@ruiyang2015
Copy link
Author

we also tested by update the Bq Lib retry func to add the 503 error in, and it can reduce the failure rate:
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/bigquery/bigquery.go#L160
return e.Code==503 || e.Code == http.StatusBadGateway || reason == "backendError" || reason == "rateLimitExceeded"

@jba
Copy link
Contributor

jba commented May 1, 2018

I added 503. But I think you should implement throttling on your end to handle these errors, which occur because you're overloading the backend.

gcf-owl-bot bot added a commit that referenced this issue Jan 30, 2025
@gcf-owl-bot
feat: Update Compute Engine API to revision 20250119 (#981)

Verified

This commit was signed with the committer’s verified signature.
bigdaz Daz DeBoer
GPG key ID: DD6B9F0B06683D5D
Verified
Learn about vigilant mode
2628da7 
Source-Link: googleapis/googleapis@2e899ee

Source-Link: googleapis/googleapis-gen@f3e46d9
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjNlNDZkOTNjYWIxNmQ3MTU1Mjg1OGRlYTM5OWQyZDE4OTZmMGI2ZiJ9
@gcf-owl-bot gcf-owl-bot bot mentioned this issue Jan 30, 2025
Merged
1 task
gcf-merge-on-green bot pushed a commit that referenced this issue Jan 31, 2025
@gcf-owl-bot
docs(monitoring/apiv3): remove extra fenced code block markers (#11535)
59fe58a 
- [ ] Regenerate this pull request now.

feat(monitoring/apiv3): add filter field to snooze proto

PiperOrigin-RevId: 721410619

Source-Link: https://togithub.com/googleapis/googleapis/commit/4d95834e9db8ee827f0e7846b175a3aeaf92f9ef

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/6f3fbcbbcd2a61dd72b0144ecd7043a9973d39e1
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNmYzZmJjYmJjZDJhNjFkZDcyYjAxNDRlY2Q3MDQzYTk5NzNkMzllMSJ9
BEGIN_NESTED_COMMIT
feat(netapp): add support for Quota Rule apis
Clients can now use quota rules for limiting the maximum usage on a volume by individual users or groups. They now also have an option for setting the default user and default group quota rule. Introducing CreateQuotaRule, UpdateQuotaRule, DeleteQuotaRule, GetQuotaRule and ListQuotaRules apis.

feat(netapp): add ipAddress field to MountOption
PiperOrigin-RevId: 721387965

Source-Link: https://togithub.com/googleapis/googleapis/commit/d0cdaf8097981ede7d8132e768229e02416d9974

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/8c7cb0dbb1907c2ceca4fb332977a702fa24583d
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOGM3Y2IwZGJiMTkwN2MyY2VjYTRmYjMzMjk3N2E3MDJmYTI0NTgzZCJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(pubsub): add support for message transforms to Topic and Subscription
docs(pubsub): Fix link for AnalyticsHubSubscriptionInfo

PiperOrigin-RevId: 721335502

Source-Link: https://togithub.com/googleapis/googleapis/commit/616127016cd4e657a1e177462539f581fbca5c7a

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/aeb0bb9f104a35e3bd45b7fc8de8ef27987a6e58
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYWViMGJiOWYxMDRhMzVlM2JkNDViN2ZjOGRlOGVmMjc5ODdhNmU1OCJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(spanner/admin/database): add AddSplitPoints API
PiperOrigin-RevId: 721248606

Source-Link: https://togithub.com/googleapis/googleapis/commit/d57f2c114b2d1d3db7fa71a1333d72129f8fd1ae

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/c2418f305f5002010264d2533fbcb7a900353499
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzI0MThmMzA1ZjUwMDIwMTAyNjRkMjUzM2ZiY2I3YTkwMDM1MzQ5OSJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(compute): Update Compute Engine API to revision 20250119 (#981)
Source-Link: https://togithub.com/googleapis/googleapis/commit/2e899eed4464394e6298e3c4cfd4a40d83a6b1ba

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/f3e46d93cab16d71552858dea399d2d1896f0b6f
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjNlNDZkOTNjYWIxNmQ3MTU1Mjg1OGRlYTM5OWQyZDE4OTZmMGI2ZiJ9
END_NESTED_COMMIT
@release-please release-please bot mentioned this issue Jan 31, 2025
Merged
gcf-merge-on-green bot pushed a commit that referenced this issue Feb 5, 2025
@release-please
chore: release main (#11550)
b1e3ce6 
🤖 I have created a release *beep* *boop*
---


<details><summary>aiplatform: 1.72.0</summary>

## [1.72.0](https://togithub.com/googleapis/google-cloud-go/compare/aiplatform/v1.71.0...aiplatform/v1.72.0) (2025-02-05)


### Features

* **aiplatform:** Add rag_files_count to RagCorpus to count number of associated files ([678944b](https://togithub.com/googleapis/google-cloud-go/commit/678944b30e389781687209caf3e3b9d35739a6f0))
</details>

<details><summary>compute: 1.33.0</summary>

## [1.33.0](https://togithub.com/googleapis/google-cloud-go/compare/compute/v1.32.0...compute/v1.33.0) (2025-02-05)


### Features

* **compute:** Update Compute Engine API to revision 20250119 ([#981](https://togithub.com/googleapis/google-cloud-go/issues/981)) ([59fe58a](https://togithub.com/googleapis/google-cloud-go/commit/59fe58aba61abf69bfb7549c0a03b21bdb4b8b2f))
</details>

<details><summary>monitoring: 1.24.0</summary>

## [1.24.0](https://togithub.com/googleapis/google-cloud-go/compare/monitoring/v1.23.0...monitoring/v1.24.0) (2025-02-05)


### Features

* **monitoring/apiv3:** Add filter field to snooze proto ([59fe58a](https://togithub.com/googleapis/google-cloud-go/commit/59fe58aba61abf69bfb7549c0a03b21bdb4b8b2f))


### Documentation

* **monitoring/apiv3:** Remove extra fenced code block markers ([#11535](https://togithub.com/googleapis/google-cloud-go/issues/11535)) ([59fe58a](https://togithub.com/googleapis/google-cloud-go/commit/59fe58aba61abf69bfb7549c0a03b21bdb4b8b2f))
</details>

<details><summary>netapp: 1.7.0</summary>

## [1.7.0](https://togithub.com/googleapis/google-cloud-go/compare/netapp/v1.6.0...netapp/v1.7.0) (2025-02-05)


### Features

* **netapp:** Add ipAddress field to MountOption ([59fe58a](https://togithub.com/googleapis/google-cloud-go/commit/59fe58aba61abf69bfb7549c0a03b21bdb4b8b2f))
* **netapp:** Add support for Quota Rule apis ([59fe58a](https://togithub.com/googleapis/google-cloud-go/commit/59fe58aba61abf69bfb7549c0a03b21bdb4b8b2f))
</details>

<details><summary>run: 1.9.0</summary>

## [1.9.0](https://togithub.com/googleapis/google-cloud-go/compare/run/v1.8.1...run/v1.9.0) (2025-02-05)


### Features

* **run:** Add Base Image URI to Container ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))
* **run:** Add BuildConfig to Services for configuring Run functions ([#11559](https://togithub.com/googleapis/google-cloud-go/issues/11559)) ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))
* **run:** Add BuildInfo to Revision for displaying BuildConfig used for a specific revision deployment ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))
* **run:** Add creator field to Execution ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))
* **run:** Add project_descriptor to BuildspackBuild ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))


### Documentation

* **run:** Some typos were fixed and formatting changed ([72046f5](https://togithub.com/googleapis/google-cloud-go/commit/72046f59d15f19aa901cd43de23b42ded1860745))
</details>

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jba jba

Labels
api: bigquery Issues related to the BigQuery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jeanbza @ruiyang2015 @jba