feat: add support for Proto Columns (#2779)

* feat: Support for Proto Messages & Enums (#2155) * feat: Importing Proto Changes Commit will be reverted, once PROTO changes are available publicly. * feat: Proto Message Implementation * feat: Adding support for enum * feat: Code refactoring Adding default implementation for newly added methods ByteArray compatability changes for Proto Messages * docs: Adding Java docs for all the newly added methods. * test: Sample Proto & Generated classes for unit test * feat: Adding bytes/proto & int64/enum compatability Adding Additional check for ChecksumResultSet * test: Adding unit tests * test: Adding unit tests for ValueBinder.java * feat: refactoring to add support for getValue & other minor changes * feat: Minor refactoring 1. Adding docs and formatting the code. 2. Adding additional methods for enum and message which accepts descriptors. * feat: Adding bytes/message & int64/enum compatability in Value * refactor: Minor refactoring * feat: Adding Proto Array Implementation * test: Implementing unit tests for array of protos and enums * refactor: adding clirr ignores * feat: Adding support for enum as Primary Key * feat: Code Review Changes, minor refactoring and adding docs * feat: Addressing review comments -Modified Docs/Comments -Minor Refactoring * refactor: Using Column instead of column to avoid test failures * feat: Minor refactoring -code review comments -adding function docs * samples: Adding samples for updating & querying Proto messages & enums (#2211) * samples: Adding samples for updating & querying Proto messages & enums * style: linting * style: linting * docs: Adding function and class doc * test: Proto Column Integration tests (#2212) * test: Adding Integration tests for Proto Messages & Enums * test: Adding additional test for Parameterized Queries, Primary Keys & Invalid Wire type errors. * style: Formatting * style: Formatting * test: Updating instance and db name * test: Adding inter compatability check while writing data * Configured jitpack.yml to use OpenJDK 11 (#2218) Co-authored-by: Pavol Juhos <pjuhos@google.com> * feat: add support for Proto Columns DDL (#2277) * feat: add code changes and tests for Proto columns DDL support * feat: add auto generated code * feat: code changes and tests for Proto columns DDL support * feat: add descriptors file * feat: code refactoring * feat: Integration tests and code refactoring * feat: code refactoring * feat: unit tests and clirr differences * feat: lint changes * feat: code refactor * feat: code refactoring * feat: code refactoring * feat: code refactoring * feat: add java docs to new methods * feat: lint formatting * feat: lint formatting changes * feat: lint formatting * feat: lint formatting * feat: test exception cases * feat: code refactoring * feat: add java docs and refactoring * feat: add java docs * feat: java docs refactor * feat: remove overload method setProtoDescriptors that accepts file path as input to avoid unexpected issues * feat: remove updateDdl method overload to update proto descriptor * teat: update pom file to run tests on cloud-devel region temporarily to validate main branch update * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: revert host changes in pom.xml file * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): revert autogenerated code * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): remove samples * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): remove clirr * feat(spanner): skip emulator test * feat(spanner): clirr * feat(spanner): fix javadoc * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): fix javadoc * feat(spanner): fix javadoc * feat(spanner): fix javadoc * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): fix emulator skip * build: ignore all changes in v1 package * feat(spanner): add optimizations to deserialize proto messages * feat(spanner): remove TODO * feat(spanner): remove TODO --------- Co-authored-by: Gaurav Purohit <gauravpurohit@google.com> Co-authored-by: Pavol Juhos <pjuhos@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Knut Olav Løite <koloite@gmail.com>
googleapis · Jan 26, 2024 · 30d37dd · 30d37dd
1 parent 95f064f
commit 30d37dd
Show file tree

Hide file tree

Showing 38 changed files with 3,945 additions and 117 deletions.
diff --git a/google-cloud-spanner/clirr-ignored-differences.xml b/google-cloud-spanner/clirr-ignored-differences.xml
@@ -192,6 +192,46 @@
     <className>com/google/cloud/spanner/StructReader</className>
     <method>java.util.List getPgJsonbList(java.lang.String)</method>
   </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>com.google.protobuf.ProtocolMessageEnum getProtoEnum(int, java.util.function.Function)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>com.google.protobuf.ProtocolMessageEnum getProtoEnum(java.lang.String, java.util.function.Function)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>com.google.protobuf.AbstractMessage getProtoMessage(int, com.google.protobuf.AbstractMessage)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>com.google.protobuf.AbstractMessage getProtoMessage(java.lang.String, com.google.protobuf.AbstractMessage)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>java.util.List getProtoEnumList(int, java.util.function.Function)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>java.util.List getProtoEnumList(java.lang.String, java.util.function.Function)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>java.util.List getProtoMessageList(int, com.google.protobuf.AbstractMessage)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/StructReader</className>
+    <method>java.util.List getProtoMessageList(java.lang.String, com.google.protobuf.AbstractMessage)</method>
+  </difference>
   <difference>
     <differenceType>7012</differenceType>
     <className>com/google/cloud/spanner/BatchClient</className>
@@ -222,6 +262,38 @@
     <className>com/google/cloud/spanner/connection/Connection</className>
     <method>com.google.cloud.spanner.ResultSet analyzeUpdateStatement(com.google.cloud.spanner.Statement, com.google.cloud.spanner.ReadContext$QueryAnalyzeMode, com.google.cloud.spanner.Options$UpdateOption[])</method>
   </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/DatabaseAdminClient</className>
+    <method>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse getDatabaseDdlResponse(java.lang.String, java.lang.String)</method>
+  </difference>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/spanner/DatabaseAdminClient</className>
+    <method>com.google.api.gax.longrunning.OperationFuture updateDatabaseDdl(com.google.cloud.spanner.Database, java.lang.Iterable, java.lang.String)</method>
+  </difference>
+  <difference>
+    <differenceType>7013</differenceType>
+    <className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
+    <method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(byte[])</method>
+  </difference>
+  <difference>
+    <differenceType>7013</differenceType>
+    <className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
+    <method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(java.io.InputStream)</method>
+  </difference>
+  <difference>
+    <differenceType>7013</differenceType>
+    <className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
+    <method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(java.lang.String)</method>
+  </difference>
+  <difference>
+    <differenceType>7006</differenceType>
+    <className>com/google/cloud/spanner/spi/v1/GapicSpannerRpc</className>
+    <method>java.util.List getDatabaseDdl(java.lang.String)</method>
+    <from>java.util.List</from>
+    <to>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse</to>
+  </difference>
   <difference>
     <differenceType>7004</differenceType>
     <className>com/google/cloud/spanner/spi/v1/GapicSpannerRpc</className>
@@ -277,6 +349,13 @@
     <method>com.google.spanner.v1.ResultSet executeQuery(com.google.spanner.v1.ExecuteSqlRequest, java.util.Map)</method>
     <to>com.google.cloud.spanner.spi.v1.SpannerRpc$StreamingCall</to>
   </difference>
+  <difference>
+    <differenceType>7006</differenceType>
+    <className>com/google/cloud/spanner/spi/v1/SpannerRpc</className>
+    <method>java.util.List getDatabaseDdl(java.lang.String)</method>
+    <from>java.util.List</from>
+    <to>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse</to>
+  </difference>
   <difference>
     <differenceType>7004</differenceType>
     <className>com/google/cloud/spanner/spi/v1/SpannerRpc</className>

diff --git a/google-cloud-spanner/pom.xml b/google-cloud-spanner/pom.xml
@@ -134,6 +134,15 @@
         <groupId>org.codehaus.mojo</groupId>
         <artifactId>flatten-maven-plugin</artifactId>
       </plugin>
+
+      <plugin>
+        <groupId>org.codehaus.mojo</groupId>
+        <artifactId>clirr-maven-plugin</artifactId>
+        <configuration>
+          <!-- Exclude (public) classes in this internal package -->
+          <excludes>com/google/cloud/spanner/spi/v1/**</excludes>
+        </configuration>
+      </plugin>
     </plugins>
     <pluginManagement>
       <plugins>

diff --git a/google-cloud-spanner/src/main/java/com/google/cloud/spanner/AbstractResultSet.java b/google-cloud-spanner/src/main/java/com/google/cloud/spanner/AbstractResultSet.java
@@ -39,10 +39,13 @@
 import com.google.common.collect.AbstractIterator;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Lists;
+import com.google.common.io.CharSource;
 import com.google.common.util.concurrent.Uninterruptibles;
+import com.google.protobuf.AbstractMessage;
 import com.google.protobuf.ByteString;
 import com.google.protobuf.ListValue;
 import com.google.protobuf.NullValue;
+import com.google.protobuf.ProtocolMessageEnum;
 import com.google.protobuf.Value.KindCase;
 import com.google.spanner.v1.PartialResultSet;
 import com.google.spanner.v1.ResultSetMetadata;
@@ -58,6 +61,7 @@
 import java.io.IOException;
 import java.io.Serializable;
 import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
 import java.util.AbstractList;
 import java.util.ArrayList;
 import java.util.Base64;
@@ -73,6 +77,7 @@
 import java.util.concurrent.Executor;
 import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
 import java.util.logging.Level;
 import java.util.logging.Logger;
 import java.util.stream.Collectors;
@@ -477,6 +482,14 @@ private Object writeReplace() {
           case JSON:
             builder.set(fieldName).to(Value.json((String) value));
             break;
+          case PROTO:
+            builder
+                .set(fieldName)
+                .to(Value.protoMessage((ByteArray) value, fieldType.getProtoTypeFqn()));
+            break;
+          case ENUM:
+            builder.set(fieldName).to(Value.protoEnum((Long) value, fieldType.getProtoTypeFqn()));
+            break;
           case PG_JSONB:
             builder.set(fieldName).to(Value.pgJsonb((String) value));
             break;
@@ -500,6 +513,7 @@ private Object writeReplace() {
                 builder.set(fieldName).toBoolArray((Iterable<Boolean>) value);
                 break;
               case INT64:
+              case ENUM:
                 builder.set(fieldName).toInt64Array((Iterable<Long>) value);
                 break;
               case FLOAT64:
@@ -521,6 +535,7 @@ private Object writeReplace() {
                 builder.set(fieldName).toPgJsonbArray((Iterable<String>) value);
                 break;
               case BYTES:
+              case PROTO:
                 builder
                     .set(fieldName)
                     .toBytesArrayFromBase64(
@@ -596,6 +611,7 @@ private static Object decodeValue(Type fieldType, com.google.protobuf.Value prot
           checkType(fieldType, proto, KindCase.BOOL_VALUE);
           return proto.getBoolValue();
         case INT64:
+        case ENUM:
           checkType(fieldType, proto, KindCase.STRING_VALUE);
           return Long.parseLong(proto.getStringValue());
         case FLOAT64:
@@ -610,6 +626,7 @@ private static Object decodeValue(Type fieldType, com.google.protobuf.Value prot
           checkType(fieldType, proto, KindCase.STRING_VALUE);
           return proto.getStringValue();
         case BYTES:
+        case PROTO:
           checkType(fieldType, proto, KindCase.STRING_VALUE);
           return new LazyByteArray(proto.getStringValue());
         case TIMESTAMP:
@@ -649,7 +666,8 @@ private static Struct decodeStructValue(Type structType, ListValue structValue)
     static Object decodeArrayValue(Type elementType, ListValue listValue) {
       switch (elementType.getCode()) {
         case INT64:
-          // For int64/float64 types, use custom containers.  These avoid wrapper object
+        case ENUM:
+          // For int64/float64/enum types, use custom containers.  These avoid wrapper object
           // creation for non-null arrays.
           return new Int64Array(listValue);
         case FLOAT64:
@@ -664,6 +682,7 @@ static Object decodeArrayValue(Type elementType, ListValue listValue) {
         case TIMESTAMP:
         case DATE:
         case STRUCT:
+        case PROTO:
           return Lists.transform(
               listValue.getValuesList(), input -> decodeValue(elementType, input));
         default:
@@ -699,6 +718,35 @@ public boolean isNull(int columnIndex) {
       return rowData.get(columnIndex) == null;
     }
 
+    @Override
+    protected <T extends AbstractMessage> T getProtoMessageInternal(int columnIndex, T message) {
+      Preconditions.checkNotNull(
+          message,
+          "Proto message may not be null. Use MyProtoClass.getDefaultInstance() as a parameter value.");
+      try {
+        return (T)
+            message
+                .toBuilder()
+                .mergeFrom(
+                    Base64.getDecoder()
+                        .wrap(
+                            CharSource.wrap(((LazyByteArray) rowData.get(columnIndex)).base64String)
+                                .asByteSource(StandardCharsets.UTF_8)
+                                .openStream()))
+                .build();
+      } catch (IOException ioException) {
+        throw SpannerExceptionFactory.asSpannerException(ioException);
+      }
+    }
+
+    @Override
+    protected <T extends ProtocolMessageEnum> T getProtoEnumInternal(
+        int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
+      Preconditions.checkNotNull(
+          method, "Method may not be null. Use 'MyProtoEnum::forNumber' as a parameter value.");
+      return (T) method.apply((int) getLongInternal(columnIndex));
+    }
+
     @Override
     protected boolean getBooleanInternal(int columnIndex) {
       return (Boolean) rowData.get(columnIndex);
@@ -768,6 +816,8 @@ protected Value getValueInternal(int columnIndex) {
           return Value.bool(isNull ? null : getBooleanInternal(columnIndex));
         case INT64:
           return Value.int64(isNull ? null : getLongInternal(columnIndex));
+        case ENUM:
+          return Value.protoEnum(getLongInternal(columnIndex), columnType.getProtoTypeFqn());
         case NUMERIC:
           return Value.numeric(isNull ? null : getBigDecimalInternal(columnIndex));
         case PG_NUMERIC:
@@ -782,6 +832,8 @@ protected Value getValueInternal(int columnIndex) {
           return Value.pgJsonb(isNull ? null : getPgJsonbInternal(columnIndex));
         case BYTES:
           return Value.internalBytes(isNull ? null : getLazyBytesInternal(columnIndex));
+        case PROTO:
+          return Value.protoMessage(getBytesInternal(columnIndex), columnType.getProtoTypeFqn());
         case TIMESTAMP:
           return Value.timestamp(isNull ? null : getTimestampInternal(columnIndex));
         case DATE:
@@ -812,6 +864,12 @@ protected Value getValueInternal(int columnIndex) {
               return Value.pgJsonbArray(isNull ? null : getPgJsonbListInternal(columnIndex));
             case BYTES:
               return Value.bytesArray(isNull ? null : getBytesListInternal(columnIndex));
+            case PROTO:
+              return Value.protoMessageArray(
+                  isNull ? null : getBytesListInternal(columnIndex), elementType.getProtoTypeFqn());
+            case ENUM:
+              return Value.protoEnumArray(
+                  isNull ? null : getLongListInternal(columnIndex), elementType.getProtoTypeFqn());
             case TIMESTAMP:
               return Value.timestampArray(isNull ? null : getTimestampListInternal(columnIndex));
             case DATE:
@@ -891,6 +949,61 @@ protected List<String> getJsonListInternal(int columnIndex) {
       return Collections.unmodifiableList((List<String>) rowData.get(columnIndex));
     }
 
+    @Override
+    @SuppressWarnings("unchecked") // We know ARRAY<PROTO> produces a List<ByteArray>.
+    protected <T extends AbstractMessage> List<T> getProtoMessageListInternal(
+        int columnIndex, T message) {
+      Preconditions.checkNotNull(
+          message,
+          "Proto message may not be null. Use MyProtoClass.getDefaultInstance() as a parameter value.");
+
+      List<LazyByteArray> bytesArray = (List<LazyByteArray>) rowData.get(columnIndex);
+
+      try {
+        List<T> protoMessagesList = new ArrayList<>(bytesArray.size());
+        for (LazyByteArray protoMessageBytes : bytesArray) {
+          if (protoMessageBytes == null) {
+            protoMessagesList.add(null);
+          } else {
+            protoMessagesList.add(
+                (T)
+                    message
+                        .toBuilder()
+                        .mergeFrom(
+                            Base64.getDecoder()
+                                .wrap(
+                                    CharSource.wrap(protoMessageBytes.base64String)
+                                        .asByteSource(StandardCharsets.UTF_8)
+                                        .openStream()))
+                        .build());
+          }
+        }
+        return protoMessagesList;
+      } catch (IOException ioException) {
+        throw SpannerExceptionFactory.asSpannerException(ioException);
+      }
+    }
+
+    @Override
+    @SuppressWarnings("unchecked") // We know ARRAY<ENUM> produces a List<Long>.
+    protected <T extends ProtocolMessageEnum> List<T> getProtoEnumListInternal(
+        int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
+      Preconditions.checkNotNull(
+          method, "Method may not be null. Use 'MyProtoEnum::forNumber' as a parameter value.");
+
+      List<Long> enumIntArray = (List<Long>) rowData.get(columnIndex);
+      List<T> protoEnumList = new ArrayList<>(enumIntArray.size());
+      for (Long enumIntValue : enumIntArray) {
+        if (enumIntValue == null) {
+          protoEnumList.add(null);
+        } else {
+          protoEnumList.add((T) method.apply(enumIntValue.intValue()));
+        }
+      }
+
+      return protoEnumList;
+    }
+
     @Override
     @SuppressWarnings("unchecked") // We know ARRAY<JSONB> produces a List<String>.
     protected List<String> getPgJsonbListInternal(int columnIndex) {
@@ -1489,6 +1602,17 @@ protected String getStringInternal(int columnIndex) {
     return currRow().getStringInternal(columnIndex);
   }
 
+  @Override
+  protected <T extends AbstractMessage> T getProtoMessageInternal(int columnIndex, T message) {
+    return currRow().getProtoMessageInternal(columnIndex, message);
+  }
+
+  @Override
+  protected <T extends ProtocolMessageEnum> T getProtoEnumInternal(
+      int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
+    return currRow().getProtoEnumInternal(columnIndex, method);
+  }
+
   @Override
   protected String getJsonInternal(int columnIndex) {
     return currRow().getJsonInternal(columnIndex);
@@ -1574,6 +1698,18 @@ protected List<ByteArray> getBytesListInternal(int columnIndex) {
     return currRow().getBytesListInternal(columnIndex);
   }
 
+  @Override
+  protected <T extends AbstractMessage> List<T> getProtoMessageListInternal(
+      int columnIndex, T message) {
+    return currRow().getProtoMessageListInternal(columnIndex, message);
+  }
+
+  @Override
+  protected <T extends ProtocolMessageEnum> List<T> getProtoEnumListInternal(
+      int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
+    return currRow().getProtoEnumListInternal(columnIndex, method);
+  }
+
   @Override
   protected List<Timestamp> getTimestampListInternal(int columnIndex) {
     return currRow().getTimestampListInternal(columnIndex);