Retry didn't occurred

[nitishxp]

Running on Python 39 Cloud function runtime

Lib used
google-cloud-storage==2.1.0
google-auth==1.35.0

("{'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}: None", {'error': {'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}}) Traceback (most recent call last): File "/workspace/common/common.py", line 125, in wrapper response = wrapped_func(*args, **kwargs) File "/workspace/main.py", line 180, in fn tenant_services.activate_client(gcs=True, stage_bucket=True) File "/workspace/common/services.py", line 188, in activate_client self.stage_bucket: storage.Bucket = gcs.get_bucket(<bucket_name>) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/storage/client.py", line 780, in get_bucket bucket.reload( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/storage/bucket.py", line 1029, in reload super(Bucket, self).reload( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/storage/_helpers.py", line 232, in reload api_response = client._get_resource( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/storage/client.py", line 364, in _get_resource return self._connection.api_request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/storage/_http.py", line 80, in api_request return call() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 469, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 333, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 371, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/auth/transport/requests.py", line 476, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/auth/credentials.py", line 133, in before_request self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/service_account.py", line 408, in refresh access_token, expiry, _ = _client.jwt_grant( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py", line 193, in jwt_grant response_data = _token_endpoint_request(request, token_uri, body) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py", line 165, in _token_endpoint_request _handle_error_response(response_data) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response raise exceptions.RefreshError(error_details, response_data) google.auth.exceptions.RefreshError: ("{'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}: None", {'error': {'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}})

[andrewsg]

It looks like your auth request failed. I'm sorry, but we don't have the capability in this library to wrap auth requests in retries. I can see you've already opened a bug against the auth library so please pursue the issue there for now. Thanks!

[di]

The auth library has since added support for retries in https://pypi.org/project/google-auth/2.12.0/ via googleapis/google-auth-library-python#1113, should this be reopened?

[andrewsg]

@di That's good to know! But, why reopen this bug? Is there something the Storage library needs to do to enable retries in auth?

[di]

I'm not sure! I just noticed that it's available, not sure if it happens by default or if clients need to adopt it.

(I am experiencing an issue where it appears that retries for auth aren't happening with this library, but I haven't pinpointed why yet).

[supreme-core]

It looks like your auth request failed. I'm sorry, but we don't have the capability in this library to wrap auth requests in retries. I can see you've already opened a bug against the auth library so please pursue the issue there for now. Thanks!

I am new to the API, I apologize in advance if the question sounds stupid.

Is there a way to include additional exception handling not readily available from the API. For example the case mentioned above?

Or a catch-all exception handler to handle unknown cases to prevent system crashes.