Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: update JsonStreamWriterBuilder comment and update sample to use the latest schema retrieval support #1902

Merged
merged 7 commits into from Dec 7, 2022
Merged

fix: update JsonStreamWriterBuilder comment and update sample to use the latest schema retrieval support #1902

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
67fb097
fix: update sample to use the latest schema retrival support
yirutang Dec 2, 2022
8cf2749
.
yirutang Dec 2, 2022
fdd8ef1
.
yirutang Dec 2, 2022
207501b
.
yirutang Dec 2, 2022
9adb96b
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Dec 2, 2022
6e1894e
.
yirutang Dec 7, 2022
d1971ee
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Dec 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions ...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java
View file
Open in desktop
Expand Up @@ -320,6 +320,11 @@ private void setStreamWriterSettings(
* newBuilder that constructs a JsonStreamWriter builder with BigQuery client being initialized by
* StreamWriter by default.
*
* <p>The table schema passed in will be updated automatically when there is a schema update
* event. When used for Writer creation, it should be the latest schema. So when you are trying to
* reuse a stream, you should use Builder newBuilder( String streamOrTableName,
* BigQueryWriteClient client) instead, so the created Writer will be based on a fresh schema.
*
* @param streamOrTableName name of the stream that must follow
* "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+" or table name
* "projects/[^/]+/datasets/[^/]+/tables/[^/]+"
Expand All @@ -336,6 +341,11 @@ public static Builder newBuilder(String streamOrTableName, TableSchema tableSche
/**
* newBuilder that constructs a JsonStreamWriter builder.
*
* <p>The table schema passed in will be updated automatically when there is a schema update
* event. When used for Writer creation, it should be the latest schema. So when you are trying to
* reuse a stream, you should use Builder newBuilder( String streamOrTableName,
* BigQueryWriteClient client) instead, so the created Writer will be based on a fresh schema.
*
* @param streamOrTableName name of the stream that must follow
* "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+"
* @param tableSchema The schema of the table when the stream was created, which is passed back
Expand Down
13 changes: 3 additions & 10 deletions samples/snippets/src/main/java/com/example/bigquerystorage/WriteToDefaultStream.java
View file
Open in desktop
Expand Up @@ -24,16 +24,14 @@
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.Schema;
import com.google.cloud.bigquery.Table;
import com.google.cloud.bigquery.TableResult;
import com.google.cloud.bigquery.storage.v1.AppendRowsResponse;
import com.google.cloud.bigquery.storage.v1.BigQueryWriteClient;
import com.google.cloud.bigquery.storage.v1.Exceptions;
import com.google.cloud.bigquery.storage.v1.Exceptions.AppendSerializtionError;
import com.google.cloud.bigquery.storage.v1.Exceptions.StorageException;
import com.google.cloud.bigquery.storage.v1.JsonStreamWriter;
import com.google.cloud.bigquery.storage.v1.TableName;
import com.google.cloud.bigquery.storage.v1.TableSchema;
import com.google.common.collect.ImmutableList;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.protobuf.Descriptors.DescriptorValidationException;
Expand Down Expand Up @@ -146,17 +144,12 @@ private static class DataWriter {

public void initialize(TableName parentTable)
throws DescriptorValidationException, IOException, InterruptedException {
// Retrive table schema information.
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
Table table = bigquery.getTable(parentTable.getDataset(), parentTable.getTable());
Schema schema = table.getDefinition().getSchema();
TableSchema tableSchema = BqToBqStorageSchemaConverter.convertTableSchema(schema);

// Use the JSON stream writer to send records in JSON format. Specify the table name to write
// to the default stream.
// For more information about JsonStreamWriter, see:
// https://googleapis.dev/java/google-cloud-bigquerystorage/latest/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.html
streamWriter = JsonStreamWriter.newBuilder(parentTable.toString(), tableSchema).build();
streamWriter =
JsonStreamWriter.newBuilder(parentTable.toString(), BigQueryWriteClient.create()).build();
}

public void append(AppendContext appendContext)
Expand Down