Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified parameters descriptions as per public doc #1480

Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
4734523
Modified parameters descriptions as per public doc
sharan-malyala Apr 8, 2024
fc13f52
Modified parameters for bigtable to vector embeddings template
sharan-malyala Apr 27, 2024
e8b30fd
Modified missing parameters for bigtable to vector embeddings template
sharan-malyala Apr 27, 2024
b82587a
Modified parameters descriptions for gcs to bigtable template
sharan-malyala Apr 27, 2024
26bc39c
Modified parameters descriptions for text to datastore template
sharan-malyala Apr 27, 2024
8492e30
Modified parameters descriptions for google ads to bigquery template
sharan-malyala Apr 27, 2024
584aebb
Modified parameters for PubSub avro to Bigquery template
sharan-malyala Apr 28, 2024
5eb4f7c
Modified parameters descriptions for DataStream to SQL template
sharan-malyala Apr 28, 2024
b5fd4cb
Modified parameters descriptions for PubSub to text template
sharan-malyala Apr 28, 2024
62326bf
pubsub text to java changes
sharan-malyala Apr 28, 2024
c17a793
Modified parameters for Oracle to Bigquery template
sharan-malyala Apr 29, 2024
7a4286a
Modified parameter descriptions for Bigtable change streams to Pub/Su…
sharan-malyala Apr 29, 2024
c3ab422
reverted example properties
sharan-malyala Apr 29, 2024
19257d0
reverted example
sharan-malyala Apr 29, 2024
03f0a42
Apply suggestions from code review
sharan-malyala Apr 30, 2024
57c21d1
reverted the links
sharan-malyala Apr 30, 2024
6c048c9
changed a description
sharan-malyala Apr 30, 2024
d4a8841
pull origin
sharan-malyala Apr 30, 2024
78c92bf
Apply suggestions from code review
sharan-malyala Apr 30, 2024
64443e5
Apply suggestions from code review
sharan-malyala May 1, 2024
1dc745f
modified DayPattern parameter
sharan-malyala May 3, 2024
f7e8acc
Merge branch 'main' into sharantej-dataflowTemplates
sharan-malyala May 3, 2024
b1dc28d
applied suggestions
sharan-malyala May 4, 2024
27fa668
removed optional from a description
sharan-malyala May 6, 2024
d163e07
ran spotless:apply
sharan-malyala May 6, 2024
52181f7
Apply suggestions from code review
sharan-malyala May 7, 2024
d9916c4
Merge branch 'GoogleCloudPlatform:main' into sharantej-dataflowTemplates
sharan-malyala May 7, 2024
623e467
Merge branch 'GoogleCloudPlatform:main' into sharantej-dataflowTemplates
sharan-malyala May 8, 2024
12b84a1
Merge branch 'GoogleCloudPlatform:main' into sharantej-dataflowTemplates
sharan-malyala May 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,7 @@ public interface Options extends PipelineOptions {
order = 1,
description = "Project ID",
helpText =
"The ID of the Google Cloud project of the Cloud Bigtable instance that you want to"
+ " read data from")
"The ID for the Google Cloud project that contains the Bigtable instance that you want to read data from.")
ValueProvider<String> getBigtableProjectId();

@SuppressWarnings("unused")
Expand All @@ -93,7 +92,7 @@ public interface Options extends PipelineOptions {
order = 2,
regexes = {"[a-z][a-z0-9\\-]+[a-z0-9]"},
description = "Instance ID",
helpText = "The ID of the Cloud Bigtable instance that contains the table")
helpText = "The ID of the Cloud Bigtable instance that contains the table.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableInstanceId();

@SuppressWarnings("unused")
Expand All @@ -103,7 +102,7 @@ public interface Options extends PipelineOptions {
order = 3,
regexes = {"[_a-zA-Z0-9][-_.a-zA-Z0-9]*"},
description = "Table ID",
helpText = "The ID of the Cloud Bigtable table to read")
helpText = "The ID of the Cloud Bigtable table to read.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableTableId();

@SuppressWarnings("unused")
Expand All @@ -113,7 +112,7 @@ public interface Options extends PipelineOptions {
order = 4,
optional = true,
description = "Cloud Storage directory for storing JSON files",
helpText = "The Cloud Storage path where the output JSON files can be stored.",
helpText = "The Cloud Storage path where the output JSON files are stored. For example: gs://your-bucket/your-path/.",
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
example = "gs://your-bucket/your-path/")
ValueProvider<String> getOutputDirectory();

Expand All @@ -123,7 +122,7 @@ public interface Options extends PipelineOptions {
@TemplateParameter.Text(
order = 5,
description = "JSON file prefix",
helpText = "The prefix of the JSON file name. For example, \"table1-\"")
helpText = "The prefix of the JSON filename. For example: \"table1-\". If no value is provided, defaults to \"part\".")
@Default.String("part")
ValueProvider<String> getFilenamePrefix();

Expand Down Expand Up @@ -166,7 +165,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "The byte size of the embeddings array. Can be 4 or 8.",
helpText =
"The byte size of each entry in the embeddings array. Use 4 for Float, and 8 for Double.")
"The byte size of each entry in the embeddings array. For float, use the value 4. For double, use the value 8. Defaults to: 4.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
@Default.Integer(4)
ValueProvider<Integer> getEmbeddingByteSize();

Expand All @@ -178,7 +177,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "Allow restricts mappings",
helpText =
"The comma separated fully qualified column names of the columns that should be used as the `allow` restricts, with their alias. In the format cf:col->alias.")
"The comma-separated, fully qualified column names for the columns to use as the allow restricts, with their aliases. In the format cf:col->alias.")
ValueProvider<String> getAllowRestrictsMappings();

@SuppressWarnings("unused")
Expand All @@ -189,7 +188,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "Deny restricts mappings",
helpText =
"The comma separated fully qualified column names of the columns that should be used as the `deny` restricts, with their alias. In the format cf:col->alias.")
"The comma-separated, fully qualified column names for the columns to use as the deny restricts, with their aliases. In the format cf:col->alias.")
ValueProvider<String> getDenyRestrictsMappings();

@SuppressWarnings("unused")
Expand All @@ -200,7 +199,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "Integer numeric restricts mappings",
helpText =
"The comma separated fully qualified column names of the columns that should be used as integer `numeric_restricts`, with their alias. In the format cf:col->alias.")
"The comma-separated, fully qualified column names of the columns to use as integer numeric_restricts, with their aliases. In the format cf:col->alias.")
ValueProvider<String> getIntNumericRestrictsMappings();

@SuppressWarnings("unused")
Expand All @@ -211,7 +210,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "Float numeric restricts mappings",
helpText =
"The comma separated fully qualified column names of the columns that should be used as float (4 bytes) `numeric_restricts`, with their alias. In the format cf:col->alias.")
"The comma-separated, fully qualified column names of the columns to use as float (4 bytes) numeric_restricts, with their aliases. In the format cf:col->alias.")
ValueProvider<String> getFloatNumericRestrictsMappings();

@SuppressWarnings("unused")
Expand All @@ -222,7 +221,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "Double numeric restricts mappings",
helpText =
"The comma separated fully qualified column names of the columns that should be used as double (8 bytes) `numeric_restricts`, with their alias. In the format cf:col->alias.")
"The comma-separated, fully qualified column names of the columns to use as double (8 bytes) numeric_restricts, with their aliases. In the format cf:col->alias.")
ValueProvider<String> getDoubleNumericRestrictsMappings();

@SuppressWarnings("unused")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ public interface Options extends PipelineOptions {
order = 1,
regexes = {"^[a-zA-Z0-9\\.\\-,]*$"},
description = "Cassandra Hosts",
helpText = "Comma separated value list of hostnames or ips of the Cassandra nodes.")
helpText = "The hosts of the Apache Cassandra nodes in a comma-separated list.")
ValueProvider<String> getCassandraHosts();

@SuppressWarnings("unused")
Expand All @@ -85,7 +85,7 @@ public interface Options extends PipelineOptions {
"^([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$"
},
description = "Cassandra Port",
helpText = "The port where cassandra can be reached. Defaults to 9042.")
helpText = "The TCP port to reach Apache Cassandra on the nodes (defaults to 9042).")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
@Default.Integer(9042)
ValueProvider<Integer> getCassandraPort();

Expand All @@ -96,7 +96,7 @@ public interface Options extends PipelineOptions {
order = 3,
regexes = {"^[a-zA-Z0-9][a-zA-Z0-9_]{0,47}$"},
description = "Cassandra Keyspace",
helpText = "Cassandra Keyspace where the table to be migrated can be located.")
helpText = "The Apache Cassandra keyspace where the table is located.")
ValueProvider<String> getCassandraKeyspace();

@SuppressWarnings("unused")
Expand All @@ -106,7 +106,7 @@ public interface Options extends PipelineOptions {
order = 4,
regexes = {"^[a-zA-Z][a-zA-Z0-9_]*$"},
description = "Cassandra Table",
helpText = "The name of the Cassandra table to Migrate")
helpText = "The Apache Cassandra table to be copied")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getCassandraTable();

@SuppressWarnings("unused")
Expand All @@ -115,7 +115,7 @@ public interface Options extends PipelineOptions {
@TemplateParameter.ProjectId(
order = 5,
description = "Bigtable Project ID",
helpText = "The Project ID where the target Bigtable Instance is running.")
helpText = "The Google Project ID of the Bigtable instance where the Apache Cassandra is copied.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableProjectId();

@SuppressWarnings("unused")
Expand All @@ -125,7 +125,7 @@ public interface Options extends PipelineOptions {
order = 6,
regexes = {"[a-z][a-z0-9\\-]+[a-z0-9]"},
description = "Target Bigtable Instance",
helpText = "The target Bigtable Instance where you want to write the data.")
helpText = "The Bigtable instance ID in which to copy the Apache Cassandra table.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableInstanceId();

@SuppressWarnings("unused")
Expand All @@ -135,7 +135,7 @@ public interface Options extends PipelineOptions {
order = 7,
regexes = {"[_a-zA-Z0-9][-_.a-zA-Z0-9]*"},
description = "Target Bigtable Table",
helpText = "The target Bigtable table where you want to write the data.")
helpText = "The name of the Bigtable table in which to copy the Apache Cassandra table.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableTableId();

@SuppressWarnings("unused")
Expand All @@ -146,8 +146,7 @@ public interface Options extends PipelineOptions {
optional = true,
regexes = {"[-_.a-zA-Z0-9]+"},
description = "The Default Bigtable Column Family",
helpText =
"This specifies the default column family to write data into. If no columnFamilyMapping is specified all Columns will be written into this column family. Default value is \"default\"")
helpText = "The name of the Bigtable table's column family (defaults to default).")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
@Default.String("default")
ValueProvider<String> getDefaultColumnFamily();

Expand All @@ -159,7 +158,7 @@ public interface Options extends PipelineOptions {
optional = true,
description = "The Row Key Separator",
helpText =
"All primary key fields will be appended to form your Bigtable Row Key. The rowKeySeparator allows you to specify a character separator. Default separator is '#'.")
"The separator used to build row-key (defaults to '#').")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
@Default.String("#")
ValueProvider<String> getRowKeySeparator();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ public interface Options extends PipelineOptions {
order = 1,
description = "Project ID",
helpText =
"The ID of the Google Cloud project of the Cloud Bigtable instance that you want to write data to")
"The ID of the Google Cloud project of the Cloud Bigtable instance that you want to write data to.")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableProjectId();

@SuppressWarnings("unused")
Expand All @@ -104,7 +104,7 @@ public interface Options extends PipelineOptions {
order = 3,
regexes = {"[_a-zA-Z0-9][-_.a-zA-Z0-9]*"},
description = "Table ID",
helpText = "The ID of the Cloud Bigtable table to write")
helpText = "The ID of the Cloud Bigtable table to import")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getBigtableTableId();

@SuppressWarnings("unused")
Expand All @@ -113,7 +113,7 @@ public interface Options extends PipelineOptions {
@TemplateParameter.GcsReadFile(
order = 4,
description = "Input Cloud Storage File(s)",
helpText = "The Cloud Storage location of the files you'd like to process.",
helpText = "The Cloud Storage path pattern where data is located. For example, `gs://mybucket/somefolder/prefix*`.",
example = "gs://your-bucket/your-files/*.parquet")
ValueProvider<String> getInputFilePattern();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,8 @@ public interface CommonTemplateOptions extends PipelineOptions {
optional = true,
description = "Disabled algorithms to override jdk.tls.disabledAlgorithms",
helpText =
"Comma-separated algorithms to disable. If this value is set to `none` then no algorithm is disabled. "
+ "Use with care, because the algorithms that are disabled by default are known to have either "
+ "vulnerabilities or performance issues.",
example = "SSLv3, RC4")
"Comma-separated list of algorithms to disable. If this value is set to none, no algorithm is disabled. Use this parameter with "
+ "caution, because the algorithms disabled by default might have vulnerabilities or performance issues. For example: `SSLv3, RC4`.")
@Validation.Required
ValueProvider<String> getDisabledAlgorithms();

Expand All @@ -46,11 +44,8 @@ public interface CommonTemplateOptions extends PipelineOptions {
},
description = "Extra files to stage in the workers",
helpText =
"Comma separated Cloud Storage paths or Secret Manager secrets for files to stage "
+ "in the worker. These files will be saved under the `/extra_files` directory in each "
+ "worker.",
example =
"gs://your-bucket/file.txt,projects/project-id/secrets/secret-id/versions/version-id")
"Comma-separated Cloud Storage paths or Secret Manager secrets for files to stage in the worker. These files are saved in "
+ "the /extra_files directory in each worker. For example, `gs://<my-bucket>/file.txt,projects/<project-id>/secrets/<secret-id>/versions/<version-id>`.")
@Validation.Required
ValueProvider<String> getExtraFilesToStage();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
optional = true,
description = "Shard template",
helpText =
"Defines the unique/dynamic portion of each windowed file. Recommended: use the default"
+ " (W-P-SS-of-NN). At runtime, 'W' is replaced with the window date range and 'P' is"
+ " replaced with the pane info. Repeating sequences of the letters 'S' or 'N' are"
+ " replaced with the shard number and number of shards respectively. The pipeline"
+ " assumes a single file output and will produce the text of '00-of-01' by default.",
"The shard template defines the dynamic portion of each windowed file. By default, the pipeline uses a"
+ " single shard for output to the file system within each window. This means that all data outputs into a"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " single file per window. The `outputShardTemplate` defaults `to W-P-SS-of-NN` where `W` is the window"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " date range, `P` is the pane info, `S` is the shard number, and `N` is the number of shards. In case of a single"
+ " file, the `SS-of-NN` portion of the `outputShardTemplate` is `00-of-01`.",
regexes = "^W-P-(S){1,}-of-(N){1,}$")
@Default.String("W-P-SS-of-NN")
ValueProvider<String> getOutputShardTemplate();
Expand Down Expand Up @@ -70,9 +70,9 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
regexes = {"^[^A-Za-z0-9/](y+|Y+)[^A-Za-z0-9/]$"},
description = "Custom Year Pattern to use for the output directory",
helpText =
"Pattern for formatting the year. Must be one or more of 'y' or 'Y'. Case makes no"
"Pattern for formatting the year. Must be one or more of `y` or `Y`. Case makes no"
+ " difference in the year. The pattern can be optionally wrapped by characters that"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " aren't either alphanumeric or the directory ('/') character. Defaults to 'YYYY'")
+ " aren't either alphanumeric or the directory ('/') character. Defaults to `YYYY`")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getYearPattern();

void setYearPattern(ValueProvider<String> yearPattern);
Expand All @@ -83,9 +83,9 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
regexes = {"^[^A-Za-z0-9/](M+)[^A-Za-z0-9/]$"},
description = "Custom Month Pattern to use for the output directory",
helpText =
"Pattern for formatting the month. Must be one or more of the 'M' character. The "
"Pattern for formatting the month. Must be one or more of the `M` character. The "
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ "pattern can be optionally wrapped by characters that aren't alphanumeric or the "
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ "directory ('/') character. Defaults to 'MM'")
+ "directory ('/') character. Defaults to `MM`")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getMonthPattern();

void setMonthPattern(ValueProvider<String> monthPattern);
Expand All @@ -96,10 +96,10 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
regexes = {"^[^A-Za-z0-9/](d+|D+)[^A-Za-z0-9/]$"},
description = "Custom Day Pattern to use for the output directory",
helpText =
"Pattern for formatting the day. Must be one or more of 'd' for day of month or 'D' for"
"Pattern for formatting the day. Must be one or more of `d` for day of month or `D` for"
+ " day of year. Case makes no difference in the year. The pattern can be optionally"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " wrapped by characters that aren't either alphanumeric or the directory ('/')"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " character. Defaults to 'dd'")
+ " character. Defaults to `dd`")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getDayPattern();

void setDayPattern(ValueProvider<String> dayPattern);
Expand All @@ -110,9 +110,9 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
regexes = {"^[^A-Za-z0-9/](H+)[^A-Za-z0-9/]$"},
description = "Custom Hour Pattern to use for the output directory",
helpText =
"Pattern for formatting the hour. Must be one or more of the 'H' character. The pattern"
"Pattern for formatting the hour. Must be one or more of the `H` character. The pattern"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " can be optionally wrapped by characters that aren't alphanumeric or the directory"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " ('/') character. Defaults to 'HH'")
+ " ('/') character. Defaults to `HH`")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getHourPattern();

void setHourPattern(ValueProvider<String> hourPattern);
Expand All @@ -123,9 +123,9 @@ public interface WindowedFilenamePolicyOptions extends PipelineOptions {
regexes = {"^[^A-Za-z0-9/](m+)[^A-Za-z0-9/]$"},
description = "Custom Minute Pattern to use for the output directory",
helpText =
"Pattern for formatting the minute. Must be one or more of the 'm' character. The pattern"
"Pattern for formatting the minute. Must be one or more of the `m` character. The pattern"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " can be optionally wrapped by characters that aren't alphanumeric or the directory"
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
+ " ('/') character. Defaults to 'mm'")
+ " ('/') character. Defaults to `mm`")
sharan-malyala marked this conversation as resolved.
Show resolved Hide resolved
ValueProvider<String> getMinutePattern();

void setMinutePattern(ValueProvider<String> minutePattern);
Expand Down