Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Clarified how clients should work with resumable uploads #1457

Merged
merged 3 commits into from Jun 17, 2022
Merged

docs: Clarified how clients should work with resumable uploads #1457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4093897
docs: Clarified how clients should work with resumable uploads
gcf-owl-bot[bot] Jun 16, 2022
a4fd773
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 16, 2022
a06453b
chore: update clirr-ignored-differences.xml
BenWhitehead Jun 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
36 changes: 28 additions & 8 deletions gapic-google-cloud-storage-v2/src/main/java/com/google/storage/v2/StorageClient.java
View file
Open in desktop
Expand Up @@ -2035,8 +2035,8 @@ public final ServerStreamingCallable<ReadObjectRequest, ReadObjectResponse> read
*
* @param object The object to update. The object's bucket and name fields are used to identify
* the object to update. If present, the object's generation field selects a specific revision
* of this object whose metadata should be updated. Otherwise, assumes the current, live
* version of the object.
* of this object whose metadata should be updated. Otherwise, assumes the live version of the
* object.
* @param updateMask List of fields to be updated.
* <p>To specify ALL fields, equivalent to the JSON API's "update" function, specify a single
* field with the value `&#42;`. Note: not recommended. If a new field is introduced at a
Expand Down Expand Up @@ -2125,19 +2125,39 @@ public final UnaryCallable<UpdateObjectRequest, Object> updateObjectCallable() {
* preconditions. Additionally, the final message must set 'finish_write' to true, or else it is
* an error.
*
* <p>For a resumable write, the client should instead call `StartResumableWrite()` and provide
* that method an `WriteObjectSpec.` They should then attach the returned `upload_id` to the first
* message of each following call to `Create`. If there is an error or the connection is broken
* during the resumable `Create()`, the client should check the status of the `Create()` by
* calling `QueryWriteStatus()` and continue writing from the returned `persisted_size`. This may
* be less than the amount of data the client previously sent.
* <p>For a resumable write, the client should instead call `StartResumableWrite()`, populating a
* `WriteObjectSpec` into that request. They should then attach the returned `upload_id` to the
* first message of each following call to `WriteObject`. If the stream is closed before finishing
* the upload (either explicitly by the client or due to a network error or an error response from
* the server), the client should do as follows: - Check the result Status of the stream, to
* determine if writing can be resumed on this stream or must be restarted from scratch (by
* calling `StartResumableWrite()`). The resumable errors are DEADLINE_EXCEEDED, INTERNAL, and
* UNAVAILABLE. For each case, the client should use binary exponential backoff before retrying.
* Additionally, writes can be resumed after RESOURCE_EXHAUSTED errors, but only after taking
* appropriate measures, which may include reducing aggregate send rate across clients and/or
* requesting a quota increase for your project. - If the call to `WriteObject` returns `ABORTED`,
* that indicates concurrent attempts to update the resumable write, caused either by multiple
* racing clients or by a single client where the previous request was timed out on the client
* side but nonetheless reached the server. In this case the client should take steps to prevent
* further concurrent writes (e.g., increase the timeouts, stop using more than one process to
* perform the upload, etc.), and then should follow the steps below for resuming the upload. -
* For resumable errors, the client should call `QueryWriteStatus()` and then continue writing
* from the returned `persisted_size`. This may be less than the amount of data the client
* previously sent. Note also that it is acceptable to send data starting at an offset earlier
* than the returned `persisted_size`; in this case, the service will skip data at offsets that
* were already persisted (without checking that it matches the previously written data), and
* write only the data starting from the persisted offset. This behavior can make client-side
* handling simpler in some cases.
*
* <p>The service will not view the object as complete until the client has sent a
* `WriteObjectRequest` with `finish_write` set to `true`. Sending any requests on a stream after
* sending a request with `finish_write` set to `true` will cause an error. The client
* &#42;&#42;should&#42;&#42; check the response it receives to determine how much data the
* service was able to commit and whether the service views the object as complete.
*
* <p>Attempting to resume an already finalized object will result in an OK status, with a
* WriteObjectResponse containing the finalized object's metadata.
*
* <p>Sample code:
*
* <pre>{@code
Expand Down
25 changes: 25 additions & 0 deletions gapic-google-cloud-storage-v2/src/test/java/com/google/storage/v2/StorageClientTest.java
View file
Open in desktop
Expand Up @@ -175,6 +175,7 @@ public void getBucketTest() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -198,6 +199,7 @@ public void getBucketTest() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -238,6 +240,7 @@ public void getBucketTest2() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -261,6 +264,7 @@ public void getBucketTest2() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -301,6 +305,7 @@ public void createBucketTest() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -324,6 +329,7 @@ public void createBucketTest() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -370,6 +376,7 @@ public void createBucketTest2() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -393,6 +400,7 @@ public void createBucketTest2() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -527,6 +535,7 @@ public void lockBucketRetentionPolicyTest() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -550,6 +559,7 @@ public void lockBucketRetentionPolicyTest() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -591,6 +601,7 @@ public void lockBucketRetentionPolicyTest2() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -614,6 +625,7 @@ public void lockBucketRetentionPolicyTest2() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -909,6 +921,7 @@ public void updateBucketTest() throws Exception {
Bucket.newBuilder()
.setName(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setBucketId("bucketId-1603305307")
.setEtag("etag3123477")
.setProject(ProjectName.of("[PROJECT]").toString())
.setMetageneration(1048558813)
.setLocation("location1901043637")
Expand All @@ -932,6 +945,7 @@ public void updateBucketTest() throws Exception {
.setRetentionPolicy(Bucket.RetentionPolicy.newBuilder().build())
.setIamConfig(Bucket.IamConfig.newBuilder().build())
.setSatisfiesPzs(true)
.setCustomPlacementConfig(Bucket.CustomPlacementConfig.newBuilder().build())
.setAutoclass(Bucket.Autoclass.newBuilder().build())
.build();
mockStorage.addResponse(expectedResponse);
Expand Down Expand Up @@ -1043,6 +1057,7 @@ public void getNotificationTest() throws Exception {
Notification.newBuilder()
.setName(NotificationName.of("[PROJECT]", "[BUCKET]", "[NOTIFICATION]").toString())
.setTopic("topic110546223")
.setEtag("etag3123477")
.addAllEventTypes(new ArrayList<String>())
.putAllCustomAttributes(new HashMap<String, String>())
.setObjectNamePrefix("objectNamePrefix-1978236516")
Expand Down Expand Up @@ -1086,6 +1101,7 @@ public void getNotificationTest2() throws Exception {
Notification.newBuilder()
.setName(NotificationName.of("[PROJECT]", "[BUCKET]", "[NOTIFICATION]").toString())
.setTopic("topic110546223")
.setEtag("etag3123477")
.addAllEventTypes(new ArrayList<String>())
.putAllCustomAttributes(new HashMap<String, String>())
.setObjectNamePrefix("objectNamePrefix-1978236516")
Expand Down Expand Up @@ -1129,6 +1145,7 @@ public void createNotificationTest() throws Exception {
Notification.newBuilder()
.setName(NotificationName.of("[PROJECT]", "[BUCKET]", "[NOTIFICATION]").toString())
.setTopic("topic110546223")
.setEtag("etag3123477")
.addAllEventTypes(new ArrayList<String>())
.putAllCustomAttributes(new HashMap<String, String>())
.setObjectNamePrefix("objectNamePrefix-1978236516")
Expand Down Expand Up @@ -1175,6 +1192,7 @@ public void createNotificationTest2() throws Exception {
Notification.newBuilder()
.setName(NotificationName.of("[PROJECT]", "[BUCKET]", "[NOTIFICATION]").toString())
.setTopic("topic110546223")
.setEtag("etag3123477")
.addAllEventTypes(new ArrayList<String>())
.putAllCustomAttributes(new HashMap<String, String>())
.setObjectNamePrefix("objectNamePrefix-1978236516")
Expand Down Expand Up @@ -1309,6 +1327,7 @@ public void composeObjectTest() throws Exception {
Object.newBuilder()
.setName("name3373707")
.setBucket(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setEtag("etag3123477")
.setGeneration(305703192)
.setMetageneration(1048558813)
.setStorageClass("storageClass871353277")
Expand Down Expand Up @@ -1482,6 +1501,7 @@ public void getObjectTest() throws Exception {
Object.newBuilder()
.setName("name3373707")
.setBucket(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setEtag("etag3123477")
.setGeneration(305703192)
.setMetageneration(1048558813)
.setStorageClass("storageClass871353277")
Expand Down Expand Up @@ -1550,6 +1570,7 @@ public void getObjectTest2() throws Exception {
Object.newBuilder()
.setName("name3373707")
.setBucket(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setEtag("etag3123477")
.setGeneration(305703192)
.setMetageneration(1048558813)
.setStorageClass("storageClass871353277")
Expand Down Expand Up @@ -1692,6 +1713,7 @@ public void updateObjectTest() throws Exception {
Object.newBuilder()
.setName("name3373707")
.setBucket(BucketName.of("[PROJECT]", "[BUCKET]").toString())
.setEtag("etag3123477")
.setGeneration(305703192)
.setMetageneration(1048558813)
.setStorageClass("storageClass871353277")
Expand Down Expand Up @@ -2352,6 +2374,7 @@ public void getHmacKeyTest() throws Exception {
.setState("state109757585")
.setCreateTime(Timestamp.newBuilder().build())
.setUpdateTime(Timestamp.newBuilder().build())
.setEtag("etag3123477")
.build();
mockStorage.addResponse(expectedResponse);

Expand Down Expand Up @@ -2399,6 +2422,7 @@ public void getHmacKeyTest2() throws Exception {
.setState("state109757585")
.setCreateTime(Timestamp.newBuilder().build())
.setUpdateTime(Timestamp.newBuilder().build())
.setEtag("etag3123477")
.build();
mockStorage.addResponse(expectedResponse);

Expand Down Expand Up @@ -2534,6 +2558,7 @@ public void updateHmacKeyTest() throws Exception {
.setState("state109757585")
.setCreateTime(Timestamp.newBuilder().build())
.setUpdateTime(Timestamp.newBuilder().build())
.setEtag("etag3123477")
.build();
mockStorage.addResponse(expectedResponse);

Expand Down