Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add RPO metadata settings #1105

Merged
merged 8 commits into from Jan 12, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions google-cloud-storage/clirr-ignored-differences.xml
Expand Up @@ -21,6 +21,11 @@
<method>com.google.cloud.storage.BucketInfo$Builder deleteLifecycleRules()</method>
<differenceType>7013</differenceType>
</difference>
<difference>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BenWhitehead we hit this each time we make a change to a public interface, we've been told it's to protect us from breaking changes, however we are committing breaking changes. It has gone on for years. What are your thoughts on this?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clirr is in place to primarily help us know if we are committing a change that breaks binary compatibility. The place it's most beneficial is on our generated code and our "main" apis. In this case, it's less beneficial and unfortunately BucketInfo.Builder is abstract and not annotated with @InternalExtensionOnly which would give us leeway to add methods with no guarantee of binary compatibility for out of project implementations. However, adding this annotation is itself considered a breaking change and would need to be performed at a major version.

Our policy for model classes in general has been that we consider them to generally be safe for adding new methods, unfortunately clirr isn't capable of being made aware that this is a model class and instead requires an entry defined. Unfortunately, at this point, I think we're sort of stuck having to deal with this file until either we reach a new major version when we can annotate this builder and define a general rule or we decide to refactor things and make this class not be abstract and instead simply non-final.

<className>com/google/cloud/storage/BucketInfo$Builder</className>
<method>com.google.cloud.storage.BucketInfo$Builder setRpo(com.google.cloud.storage.Rpo)</method>
<differenceType>7013</differenceType>
</difference>
<difference>
<className>com/google/cloud/storage/BucketInfo$Builder</className>
<method>com.google.cloud.storage.BucketInfo$Builder setUpdateTime(java.lang.Long)</method>
Expand Down
Expand Up @@ -577,6 +577,12 @@ public Builder deleteLifecycleRules() {
return this;
}

@Override
public Builder setRpo(Rpo rpo) {
infoBuilder.setRpo(rpo);
return this;
}

@Override
public Builder setStorageClass(StorageClass storageClass) {
infoBuilder.setStorageClass(storageClass);
Expand Down
Expand Up @@ -92,6 +92,7 @@ public com.google.api.services.storage.model.Bucket apply(BucketInfo bucketInfo)
private final List<Acl> acl;
private final List<Acl> defaultAcl;
private final String location;
private final Rpo rpo;
private final StorageClass storageClass;
private final Map<String, String> labels;
private final String defaultKmsKeyName;
Expand Down Expand Up @@ -1185,6 +1186,15 @@ public abstract static class Builder {
/** Deletes the lifecycle rules of this bucket. */
public abstract Builder deleteLifecycleRules();

/**
* Sets the bucket's Recovery Point Objective (RPO). This can only be set for a dual-region
* bucket, and determines the speed at which data will be replicated between regions. See the
* {@code Rpo} class for supported values, and <a
* href="https://cloud.google.com/storage/docs/turbo-replication">here</a> for additional
* details.
*/
public abstract Builder setRpo(Rpo rpo);

/**
* Sets the bucket's storage class. This defines how blobs in the bucket are stored and
* determines the SLA and the cost of storage. A list of supported values is available <a
Expand Down Expand Up @@ -1286,6 +1296,7 @@ static final class BuilderImpl extends Builder {
private String notFoundPage;
private List<DeleteRule> deleteRules;
private List<LifecycleRule> lifecycleRules;
private Rpo rpo;
private StorageClass storageClass;
private String location;
private String etag;
Expand Down Expand Up @@ -1317,6 +1328,7 @@ static final class BuilderImpl extends Builder {
updateTime = bucketInfo.updateTime;
metageneration = bucketInfo.metageneration;
location = bucketInfo.location;
rpo = bucketInfo.rpo;
storageClass = bucketInfo.storageClass;
cors = bucketInfo.cors;
acl = bucketInfo.acl;
Expand Down Expand Up @@ -1410,6 +1422,12 @@ public Builder deleteLifecycleRules() {
return this;
}

@Override
public Builder setRpo(Rpo rpo) {
this.rpo = rpo;
return this;
}

@Override
public Builder setStorageClass(StorageClass storageClass) {
this.storageClass = storageClass;
Expand Down Expand Up @@ -1548,6 +1566,7 @@ public BucketInfo build() {
updateTime = builder.updateTime;
metageneration = builder.metageneration;
location = builder.location;
rpo = builder.rpo;
storageClass = builder.storageClass;
cors = builder.cors;
acl = builder.acl;
Expand Down Expand Up @@ -1709,6 +1728,16 @@ public String getLocationType() {
return locationType;
}

/**
* Returns the bucket's recovery point objective (RPO). This defines how quickly data is
* replicated between regions in a dual-region bucket. Not defined for single-region buckets.
*
* @see <a href="https://cloud.google.com/storage/docs/turbo-replication"Turbo Replication"</a>
*/
public Rpo getRpo() {
return rpo;
}

/**
* Returns the bucket's storage class. This defines how blobs in the bucket are stored and
* determines the SLA and the cost of storage.
Expand Down Expand Up @@ -1879,6 +1908,9 @@ com.google.api.services.storage.model.Bucket toPb() {
if (locationType != null) {
bucketPb.setLocationType(locationType);
}
if (rpo != null) {
bucketPb.setRpo(rpo.toString());
}
if (storageClass != null) {
bucketPb.setStorageClass(storageClass.toString());
}
Expand Down Expand Up @@ -2042,6 +2074,9 @@ static BucketInfo fromPb(com.google.api.services.storage.model.Bucket bucketPb)
if (bucketPb.getLocation() != null) {
builder.setLocation(bucketPb.getLocation());
}
if (bucketPb.getRpo() != null) {
builder.setRpo(Rpo.valueOf(bucketPb.getRpo()));
}
if (bucketPb.getStorageClass() != null) {
builder.setStorageClass(StorageClass.valueOf(bucketPb.getStorageClass()));
}
Expand Down
@@ -0,0 +1,70 @@
/*
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.google.cloud.storage;

import com.google.api.core.ApiFunction;
import com.google.cloud.StringEnumType;
import com.google.cloud.StringEnumValue;

/**
* Enums for the Recovery Point Objective (RPO) of dual-region buckets, which determines how fast
* data is replicated betweens regions.
*
* @see <a
* href="https://cloud.google.com/storage/docs/turbo-replication">https://cloud.google.com/storage/docs/turbo-replication</a>
*/
public class Rpo extends StringEnumValue {
JesseLovelace marked this conversation as resolved.
Show resolved Hide resolved

private static final long serialVersionUID = -3954216195295821508L;

private Rpo(String constant) {
super(constant);
}

private static final ApiFunction<String, Rpo> CONSTRUCTOR = Rpo::new;

private static final StringEnumType<Rpo> type = new StringEnumType(Rpo.class, CONSTRUCTOR);
JesseLovelace marked this conversation as resolved.
Show resolved Hide resolved

/**
* Default recovery point objective. With this setting, there is no guarantee on the amount of
* time it takes for data to replicate between regions.
*/
public static final Rpo DEFAULT = type.createAndRegister("DEFAULT");

/**
* Turbo recovery point objective. With this setting, data in a dual-region bucket will replicate
* between regions within 15 minutes.
*/
public static final Rpo ASYNC_TURBO = type.createAndRegister("ASYNC_TURBO");

/**
* Get the Rpo for the given String constant, and throw an exception if the constant is not
* recognized.
*/
public static Rpo valueOfStrict(String constant) {
return type.valueOfStrict(constant);
}

/** Get the Rpo for the given String constant, and allow unrecognized values. */
public static Rpo valueOf(String constant) {
return type.valueOf(constant);
}

/** Return the known values for Rpo. */
public static Rpo[] values() {
return type.values();
}
}
Expand Up @@ -71,6 +71,7 @@
import com.google.cloud.storage.HttpMethod;
import com.google.cloud.storage.PostPolicyV4;
import com.google.cloud.storage.PostPolicyV4.PostFieldsV4;
import com.google.cloud.storage.Rpo;
import com.google.cloud.storage.ServiceAccount;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.Storage.BlobField;
Expand Down Expand Up @@ -3940,6 +3941,23 @@ protected Object handleInvocation(
assertArrayEquals(randStringBytes, actualData.toByteArray());
}

@Test
public void testRpoConfig() {
String rpoBucket = RemoteStorageHelper.generateBucketName();
try {
Bucket bucket =
storage.create(
BucketInfo.newBuilder(rpoBucket).setLocation("NAM4").setRpo(Rpo.ASYNC_TURBO).build());
assertEquals("ASYNC_TURBO", bucket.getRpo().toString());

bucket.toBuilder().setRpo(Rpo.DEFAULT).build().update();

assertEquals("DEFAULT", storage.get(rpoBucket).getRpo().toString());
} finally {
storage.delete(rpoBucket);
}
}

private static String randString(Random rand, int length) {
final StringBuilder sb = new StringBuilder();
while (sb.length() < length) {
Expand Down