Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add schema aware stream writer #2048

Merged
merged 3 commits into from Apr 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Expand Up @@ -49,7 +49,7 @@ If you are using Maven without BOM, add this to your dependencies:
If you are using Gradle 5.x or later, add this to your dependencies:

```Groovy
implementation platform('com.google.cloud:libraries-bom:26.11.0')
implementation platform('com.google.cloud:libraries-bom:26.12.0')

implementation 'com.google.cloud:google-cloud-bigquerystorage'
```
Expand Down
Expand Up @@ -243,8 +243,8 @@ public String getStreamName() {
}

/**
* This exception is thrown from {@link JsonStreamWriter#append()} when the client side Json to
* Proto serializtion fails. It can also be thrown by the server in case rows contains invalid
* This exception is thrown from {@link SchemaAwareStreamWriter#append()} when the client side
* Proto serialization fails. It can also be thrown by the server in case rows contains invalid
* data. The exception contains a Map of indexes of faulty rows and the corresponding error
* message.
*/
Expand Down Expand Up @@ -362,16 +362,27 @@ protected InflightBytesLimitExceededException(String writerId, long currentLimit
currentLimit);
}
}

/**
* Input Json data has unknown field to the schema of the JsonStreamWriter. User can either turn
* on IgnoreUnknownFields option on the JsonStreamWriter, or if they don't want the error to be
* ignored, they should recreate the JsonStreamWriter with the updated table schema.
* This class is replaced by a generic one. It will be removed soon. Please use {@link
* DataHasUnknownFieldException}
*/
public static final class JsonDataHasUnknownFieldException extends IllegalArgumentException {
public static final class JsonDataHasUnknownFieldException extends DataHasUnknownFieldException {
protected JsonDataHasUnknownFieldException(String jsonFieldName) {
super(jsonFieldName);
}
}
/**
* Input data object has unknown field to the schema of the SchemaAwareStreamWriter. User can
* either turn on IgnoreUnknownFields option on the SchemaAwareStreamWriter, or if they don't want
* the error to be ignored, they should recreate the SchemaAwareStreamWriter with the updated
* table schema.
*/
public static class DataHasUnknownFieldException extends IllegalArgumentException {
private final String jsonFieldName;

protected JsonDataHasUnknownFieldException(String jsonFieldName) {
super(String.format("JSONObject has fields unknown to BigQuery: %s.", jsonFieldName));
public DataHasUnknownFieldException(String jsonFieldName) {
super(String.format("The source object has fields unknown to BigQuery: %s.", jsonFieldName));
this.jsonFieldName = jsonFieldName;
}

Expand Down