Returns the pipelines Resource.
+ +Returns the transformDescriptions Resource.
+Close httplib2 connections.
+
+ computeSchema(location, body=None, x__xgafv=None)
Computes the schema for the transform. Computation from `raw_schema` will always occur if it is set. This requires that the transform supports that encoding. If no raw schema is provided and if the transform is for an IO, then this will attempt to connect to the resource using the details provided in `config` and infer the schema from that. If the transform is not an IO, is a sink that doesn't exist yet, or is a sink with no schema requirement, then this will fall back to basing the schema off the one provided in `input_schemas`. The computed schema will be validated.
listPipelines(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)
Lists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.
@@ -94,6 +102,183 @@Close httplib2 connections.+
computeSchema(location, body=None, x__xgafv=None)
+ Computes the schema for the transform. Computation from `raw_schema` will always occur if it is set. This requires that the transform supports that encoding. If no raw schema is provided and if the transform is for an IO, then this will attempt to connect to the resource using the details provided in `config` and infer the schema from that. If the transform is not an IO, is a sink that doesn't exist yet, or is a sink with no schema requirement, then this will fall back to basing the schema off the one provided in `input_schemas`. The computed schema will be validated.
+
+Args:
+ location: string, Required. The full location formatted as "projects/{your-project}/locations/{google-cloud-region}". If attempting to infer the schema from an existing Google Cloud resource, the default Data Pipelines service account for this project will be used in making requests for the resource. If the region given for "{google-cloud-region}" is different than the region where the resource is stored, then the data will be transferred to and processed in the region specified here, but it will not be persistently stored in this region. (required)
+ body: object, The request body.
+ The object takes the form of:
+
+{ # Request message for ComputeSchema
+ "config": { # A fully configured transform that can be validated. # Required. The configuration for the transform. If this is not a source, then each input with its schema must be set. It is not required to have any outputs set.
+ "config": { # Represents an Apache Beam row, though the `Any` nature of values is replaced with more concrete representations of valid values. # Configuration values provided. These must match the schema provided in the row's schema.
+ "schema": { # Holds a schema or a reference to a schema in some repository. # Required. The schema of the row's data.
+ "localSchema": { # Represents a simplified Apache Beam schema. # Schema located locally with the message.
+ "fields": [ # Fields in the schema. Every field within a schema must have a unique name.
+ { # Info for a single field in the schema.
+ "name": "A String", # Name of the field.
+ "type": { # Type info about a field. # Type info for the field.
+ "collectionElementType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # If `type` is an array or iterable, this is the type contained in that array or iterable.
+ "logicalType": { # Represents the input for creating a specified logical type. # If `type` is a logical type, this is the info for the specific logical type.
+ "enumerationType": { # Represents the Beam EnumerationType logical type. # The enum represented by this logical type.
+ "values": [ # Names of the values. The numeric value is the same as the index.
+ "A String",
+ ],
+ },
+ "fixedBytes": { # Represents the Beam FixedBytes logical type. # The fixed-size byte collection represented by this logical type.
+ "sizeBytes": 42, # Number of bytes to allocate.
+ },
+ },
+ "mapType": { # Represents a map in a schema. # If `type` is a map, this is the key and value types for that map.
+ "mapKeyType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Key type of the map. Only atomic types are supported.
+ "mapValueType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Value type of the map.
+ },
+ "nullable": True or False, # Whether or not this field is nullable.
+ "rowSchema": # Object with schema name: GoogleCloudDatapipelinesV1Schema # If `type` is a row, this is the schema of that row.
+ "type": "A String", # Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set.
+ },
+ },
+ ],
+ "referenceId": "A String", # An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository.
+ },
+ "referenceId": "A String", # The `reference_id` value of a schema in a repository.
+ },
+ "values": [ # Required. The values of this Row. A fully built row is required to hold to the schema specified by `schema`.
+ { # A single value in a row. The value set must correspond to the correct type from the row's schema.
+ "arrayValue": { # Represents an array of values. The elements can be of any type. # The array value of this field. Corresponds to TYPE_NAME_ARRAY in the schema.
+ "elements": [ # The elements of the array.
+ # Object with schema name: GoogleCloudDatapipelinesV1FieldValue
+ ],
+ },
+ "atomicValue": { # Represents a non-dividable value. # The atomic value of this field. Must correspond to the correct atomic type in the schema.
+ "booleanValue": True or False, # A boolean value.
+ "byteValue": 42, # An 8-bit signed value.
+ "bytesValue": "A String", # An array of raw bytes.
+ "datetimeValue": { # Represents civil time (or occasionally physical time). This type can represent a civil time in one of a few possible ways: * When utc_offset is set and time_zone is unset: a civil time on a calendar day with a particular offset from UTC. * When time_zone is set and utc_offset is unset: a civil time on a calendar day in a particular time zone. * When neither time_zone nor utc_offset is set: a civil time on a calendar day in local time. The date is relative to the Proleptic Gregorian Calendar. If year, month, or day are 0, the DateTime is considered not to have a specific year, month, or day respectively. This type may also be used to represent a physical time if all the date and time fields are set and either case of the `time_offset` oneof is set. Consider using `Timestamp` message for physical time instead. If your use case also would like to store the user's timezone, that can be done in another field. This type is more flexible than some applications may want. Make sure to document and validate your application's limitations. # A datetime value.
+ "day": 42, # Optional. Day of month. Must be from 1 to 31 and valid for the year and month, or 0 if specifying a datetime without a day.
+ "hours": 42, # Optional. Hours of day in 24 hour format. Should be from 0 to 23, defaults to 0 (midnight). An API may choose to allow the value "24:00:00" for scenarios like business closing time.
+ "minutes": 42, # Optional. Minutes of hour of day. Must be from 0 to 59, defaults to 0.
+ "month": 42, # Optional. Month of year. Must be from 1 to 12, or 0 if specifying a datetime without a month.
+ "nanos": 42, # Optional. Fractions of seconds in nanoseconds. Must be from 0 to 999,999,999, defaults to 0.
+ "seconds": 42, # Optional. Seconds of minutes of the time. Must normally be from 0 to 59, defaults to 0. An API may allow the value 60 if it allows leap-seconds.
+ "timeZone": { # Represents a time zone from the [IANA Time Zone Database](https://www.iana.org/time-zones). # Time zone.
+ "id": "A String", # IANA Time Zone Database time zone, e.g. "America/New_York".
+ "version": "A String", # Optional. IANA Time Zone Database version number, e.g. "2019a".
+ },
+ "utcOffset": "A String", # UTC offset. Must be whole seconds, between -18 hours and +18 hours. For example, a UTC offset of -4:00 would be represented as { seconds: -14400 }.
+ "year": 42, # Optional. Year of date. Must be from 1 to 9999, or 0 if specifying a datetime without a year.
+ },
+ "decimalValue": { # A representation of a decimal value, such as 2.5. Clients may convert values into language-native decimal formats, such as Java's BigDecimal or Python's decimal.Decimal. [BigDecimal]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/math/BigDecimal.html [decimal.Decimal]: https://docs.python.org/3/library/decimal.html # A large decimal value, equivalent to Java BigDecimal.
+ "value": "A String", # The decimal value, as a string. The string representation consists of an optional sign, `+` (`U+002B`) or `-` (`U+002D`), followed by a sequence of zero or more decimal digits ("the integer"), optionally followed by a fraction, optionally followed by an exponent. An empty string **should** be interpreted as `0`. The fraction consists of a decimal point followed by zero or more decimal digits. The string must contain at least one digit in either the integer or the fraction. The number formed by the sign, the integer and the fraction is referred to as the significand. The exponent consists of the character `e` (`U+0065`) or `E` (`U+0045`) followed by one or more decimal digits. Services **should** normalize decimal values before storing them by: - Removing an explicitly-provided `+` sign (`+2.5` -> `2.5`). - Replacing a zero-length integer value with `0` (`.5` -> `0.5`). - Coercing the exponent character to upper-case, with explicit sign (`2.5e8` -> `2.5E+8`). - Removing an explicitly-provided zero exponent (`2.5E0` -> `2.5`). Services **may** perform additional normalization based on its own needs and the internal decimal implementation selected, such as shifting the decimal point and exponent value together (example: `2.5E-1` <-> `0.25`). Additionally, services **may** preserve trailing zeroes in the fraction to indicate increased precision, but are not required to do so. Note that only the `.` character is supported to divide the integer and the fraction; `,` **should not** be supported regardless of locale. Additionally, thousand separators **should not** be supported. If a service does support them, values **must** be normalized. The ENBF grammar is: DecimalString = '' | [Sign] Significand [Exponent]; Sign = '+' | '-'; Significand = Digits '.' | [Digits] '.' Digits; Exponent = ('e' | 'E') [Sign] Digits; Digits = { '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' }; Services **should** clearly document the range of supported values, the maximum supported precision (total number of digits), and, if applicable, the scale (number of digits after the decimal point), as well as how it behaves when receiving out-of-bounds values. Services **may** choose to accept values passed as input even when the value has a higher precision or scale than the service supports, and **should** round the value to fit the supported scale. Alternatively, the service **may** error with `400 Bad Request` (`INVALID_ARGUMENT` in gRPC) if precision would be lost. Services **should** error with `400 Bad Request` (`INVALID_ARGUMENT` in gRPC) if the service receives a value outside of the supported range.
+ },
+ "doubleValue": 3.14, # A 64-bit floating point value.
+ "floatValue": 3.14, # A 32-bit floating point value.
+ "int16Value": 42, # A 16-bit signed value.
+ "int32Value": 42, # A 32-bit signed value.
+ "int64Value": "A String", # A 64-bit signed value.
+ "stringValue": "A String", # A string value.
+ },
+ "enumValue": { # Represents a selected value from an EnumerationType. # The enum value of this field. Corresponds to TYPE_NAME_LOGICAL_TYPE in the schema if that logical type represents an `EnumerationType` type.
+ "name": "A String", # Name of the enum option.
+ },
+ "fixedBytesValue": { # Represents a collection of bytes whose size is the same as the associated FixedBytes size value. # The fixed-length byte collection of this field. Corresponds to TYPE_NAME_LOGICAL_TYPE in the schema if that logical type represents a `FixedBytes` type.
+ "value": "A String", # The raw bytes. It must be exactly the size specified in the schema.
+ },
+ "iterableValue": { # Represents an iterable of values. The elements can be of any type. # The iterable value of this field. Corresponds to TYPE_NAME_ITERABLE in the schema.
+ "elements": [ # The elements of the iterable.
+ # Object with schema name: GoogleCloudDatapipelinesV1FieldValue
+ ],
+ },
+ "mapValue": { # Represents a key/value pairing. # The map value of this field. Corresponds to TYPE_NAME_MAP in the schema.
+ "entries": [ # The entries in the map.
+ { # A single entry in the map. Each entry must have a unique key.
+ "key": # Object with schema name: GoogleCloudDatapipelinesV1FieldValue # The key value. Only atomic values are supported.
+ "value": # Object with schema name: GoogleCloudDatapipelinesV1FieldValue # The value associated with the key. It may be of any type.
+ },
+ ],
+ },
+ "rowValue": # Object with schema name: GoogleCloudDatapipelinesV1Row # The row value of this field. Corresponds to TYPE_NAME_ROW in the schema. This row also holds to its own schema.
+ },
+ ],
+ },
+ "uniformResourceName": "A String", # Unique resource name of the transform. This should be the same as the equivalent `TransformDescription` value.
+ },
+ "inputSchemas": [ # Optional. In relation to the full pipeline graph, the schemas of the transforms that are used as inputs to the one for `config`. If `config` represents a transform for reading from some resource, then this should be empty. For all other transforms, at least one value must be provided.
+ { # Represents a simplified Apache Beam schema.
+ "fields": [ # Fields in the schema. Every field within a schema must have a unique name.
+ { # Info for a single field in the schema.
+ "name": "A String", # Name of the field.
+ "type": { # Type info about a field. # Type info for the field.
+ "collectionElementType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # If `type` is an array or iterable, this is the type contained in that array or iterable.
+ "logicalType": { # Represents the input for creating a specified logical type. # If `type` is a logical type, this is the info for the specific logical type.
+ "enumerationType": { # Represents the Beam EnumerationType logical type. # The enum represented by this logical type.
+ "values": [ # Names of the values. The numeric value is the same as the index.
+ "A String",
+ ],
+ },
+ "fixedBytes": { # Represents the Beam FixedBytes logical type. # The fixed-size byte collection represented by this logical type.
+ "sizeBytes": 42, # Number of bytes to allocate.
+ },
+ },
+ "mapType": { # Represents a map in a schema. # If `type` is a map, this is the key and value types for that map.
+ "mapKeyType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Key type of the map. Only atomic types are supported.
+ "mapValueType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Value type of the map.
+ },
+ "nullable": True or False, # Whether or not this field is nullable.
+ "rowSchema": # Object with schema name: GoogleCloudDatapipelinesV1Schema # If `type` is a row, this is the schema of that row.
+ "type": "A String", # Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set.
+ },
+ },
+ ],
+ "referenceId": "A String", # An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository.
+ },
+ ],
+ "rawSchema": { # The raw schema and its type. # Optional. If set, this will use the provided raw schema to compute the schema rather than connecting to any resources. Validation will still occur to make sure it is compatible with all input schemas. If the transform is an IO, the IO must support that schema type.
+ "rawSchema": "A String", # The schema.
+ "type": "A String", # The type of the schema.
+ },
+}
+
+ x__xgafv: string, V1 error format.
+ Allowed values
+ 1 - v1 error format
+ 2 - v2 error format
+
+Returns:
+ An object of the form:
+
+ { # Represents a simplified Apache Beam schema.
+ "fields": [ # Fields in the schema. Every field within a schema must have a unique name.
+ { # Info for a single field in the schema.
+ "name": "A String", # Name of the field.
+ "type": { # Type info about a field. # Type info for the field.
+ "collectionElementType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # If `type` is an array or iterable, this is the type contained in that array or iterable.
+ "logicalType": { # Represents the input for creating a specified logical type. # If `type` is a logical type, this is the info for the specific logical type.
+ "enumerationType": { # Represents the Beam EnumerationType logical type. # The enum represented by this logical type.
+ "values": [ # Names of the values. The numeric value is the same as the index.
+ "A String",
+ ],
+ },
+ "fixedBytes": { # Represents the Beam FixedBytes logical type. # The fixed-size byte collection represented by this logical type.
+ "sizeBytes": 42, # Number of bytes to allocate.
+ },
+ },
+ "mapType": { # Represents a map in a schema. # If `type` is a map, this is the key and value types for that map.
+ "mapKeyType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Key type of the map. Only atomic types are supported.
+ "mapValueType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Value type of the map.
+ },
+ "nullable": True or False, # Whether or not this field is nullable.
+ "rowSchema": # Object with schema name: GoogleCloudDatapipelinesV1Schema # If `type` is a row, this is the schema of that row.
+ "type": "A String", # Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set.
+ },
+ },
+ ],
+ "referenceId": "A String", # An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository.
+}
+listPipelines(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)
Lists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it. diff --git a/docs/dyn/datapipelines_v1.projects.locations.transformDescriptions.html b/docs/dyn/datapipelines_v1.projects.locations.transformDescriptions.html new file mode 100644 index 00000000000..9c028584fc1 --- /dev/null +++ b/docs/dyn/datapipelines_v1.projects.locations.transformDescriptions.html @@ -0,0 +1,193 @@ + + + +Data pipelines API . projects . locations . transformDescriptions
+Instance Methods
++
+batchGet(parent, names=None, x__xgafv=None)Gets transform descriptions in a batch, associated with a list of provided uniform resource names.
++
+close()Close httplib2 connections.
+ +Gets the transform description associated with the provided uniform resource name.
+Method Details
+++ +batchGet(parent, names=None, x__xgafv=None)+Gets transform descriptions in a batch, associated with a list of provided uniform resource names. + +Args: + parent: string, Required. The project and location shared by all transform descriptions being retrieved, formatted as "projects/{project}/locations/{location}". (required) + names: string, Optional. The names of the transform descriptions being retrieved, formatted as "projects/{project}/locations/{location}/transformdescriptions/{transform_description}". If no name is provided, all of the transform descriptions will be returned. (repeated) + x__xgafv: string, V1 error format. + Allowed values + 1 - v1 error format + 2 - v2 error format + +Returns: + An object of the form: + + { # Response message for BatchGetTransformDescriptions + "transformDescriptions": [ # List of requested transform descriptions. + { # Description of a schema-aware transform, which provides info on how it can be configured. + "name": "A String", # Output only. The full name of this resource formatted as: projects/{project}/locations/{location}/transformDescriptions/{transform_description} `transform_description` is the same as the `uniform_resource_name` field. + "options": { # Represents a simplified Apache Beam schema. # Available options for configuring the transform. + "fields": [ # Fields in the schema. Every field within a schema must have a unique name. + { # Info for a single field in the schema. + "name": "A String", # Name of the field. + "type": { # Type info about a field. # Type info for the field. + "collectionElementType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # If `type` is an array or iterable, this is the type contained in that array or iterable. + "logicalType": { # Represents the input for creating a specified logical type. # If `type` is a logical type, this is the info for the specific logical type. + "enumerationType": { # Represents the Beam EnumerationType logical type. # The enum represented by this logical type. + "values": [ # Names of the values. The numeric value is the same as the index. + "A String", + ], + }, + "fixedBytes": { # Represents the Beam FixedBytes logical type. # The fixed-size byte collection represented by this logical type. + "sizeBytes": 42, # Number of bytes to allocate. + }, + }, + "mapType": { # Represents a map in a schema. # If `type` is a map, this is the key and value types for that map. + "mapKeyType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Key type of the map. Only atomic types are supported. + "mapValueType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Value type of the map. + }, + "nullable": True or False, # Whether or not this field is nullable. + "rowSchema": # Object with schema name: GoogleCloudDatapipelinesV1Schema # If `type` is a row, this is the schema of that row. + "type": "A String", # Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set. + }, + }, + ], + "referenceId": "A String", # An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository. + }, + "uniformResourceName": "A String", # Unique resource name of the transform. + }, + ], +}+++ +close()+Close httplib2 connections.+++ + \ No newline at end of file diff --git a/googleapiclient/discovery_cache/documents/datapipelines.v1.json b/googleapiclient/discovery_cache/documents/datapipelines.v1.json index 81fa806a255..3fb18e04756 100644 --- a/googleapiclient/discovery_cache/documents/datapipelines.v1.json +++ b/googleapiclient/discovery_cache/documents/datapipelines.v1.json @@ -109,6 +109,34 @@ "resources": { "locations": { "methods": { + "computeSchema": { + "description": "Computes the schema for the transform. Computation from `raw_schema` will always occur if it is set. This requires that the transform supports that encoding. If no raw schema is provided and if the transform is for an IO, then this will attempt to connect to the resource using the details provided in `config` and infer the schema from that. If the transform is not an IO, is a sink that doesn't exist yet, or is a sink with no schema requirement, then this will fall back to basing the schema off the one provided in `input_schemas`. The computed schema will be validated.", + "flatPath": "v1/projects/{projectsId}/locations/{locationsId}:computeSchema", + "httpMethod": "POST", + "id": "datapipelines.projects.locations.computeSchema", + "parameterOrder": [ + "location" + ], + "parameters": { + "location": { + "description": "Required. The full location formatted as \"projects/{your-project}/locations/{google-cloud-region}\". If attempting to infer the schema from an existing Google Cloud resource, the default Data Pipelines service account for this project will be used in making requests for the resource. If the region given for \"{google-cloud-region}\" is different than the region where the resource is stored, then the data will be transferred to and processed in the region specified here, but it will not be persistently stored in this region.", + "location": "path", + "pattern": "^projects/[^/]+/locations/[^/]+$", + "required": true, + "type": "string" + } + }, + "path": "v1/{+location}:computeSchema", + "request": { + "$ref": "GoogleCloudDatapipelinesV1ComputeSchemaRequest" + }, + "response": { + "$ref": "GoogleCloudDatapipelinesV1Schema" + }, + "scopes": [ + "https://www.googleapis.com/auth/cloud-platform" + ] + }, "listPipelines": { "description": "Lists pipelines. Returns a \"FORBIDDEN\" error if the caller doesn't have permission to access it.", "flatPath": "v1/projects/{projectsId}/locations/{locationsId}", @@ -365,15 +393,198 @@ } } } + }, + "transformDescriptions": { + "methods": { + "batchGet": { + "description": "Gets transform descriptions in a batch, associated with a list of provided uniform resource names.", + "flatPath": "v1/projects/{projectsId}/locations/{locationsId}/transformDescriptions:batchGet", + "httpMethod": "GET", + "id": "datapipelines.projects.locations.transformDescriptions.batchGet", + "parameterOrder": [ + "parent" + ], + "parameters": { + "names": { + "description": "Optional. The names of the transform descriptions being retrieved, formatted as \"projects/{project}/locations/{location}/transformdescriptions/{transform_description}\". If no name is provided, all of the transform descriptions will be returned.", + "location": "query", + "repeated": true, + "type": "string" + }, + "parent": { + "description": "Required. The project and location shared by all transform descriptions being retrieved, formatted as \"projects/{project}/locations/{location}\".", + "location": "path", + "pattern": "^projects/[^/]+/locations/[^/]+$", + "required": true, + "type": "string" + } + }, + "path": "v1/{+parent}/transformDescriptions:batchGet", + "response": { + "$ref": "GoogleCloudDatapipelinesV1BatchGetTransformDescriptionsResponse" + }, + "scopes": [ + "https://www.googleapis.com/auth/cloud-platform" + ] + }, + "get": { + "description": "Gets the transform description associated with the provided uniform resource name.", + "flatPath": "v1/projects/{projectsId}/locations/{locationsId}/transformDescriptions/{transformDescriptionsId}", + "httpMethod": "GET", + "id": "datapipelines.projects.locations.transformDescriptions.get", + "parameterOrder": [ + "name" + ], + "parameters": { + "name": { + "description": "Required. The full name formatted as \"projects/{your-project}/locations/{google-cloud-region}/transformdescriptions/{uniform-resource-name}\".", + "location": "path", + "pattern": "^projects/[^/]+/locations/[^/]+/transformDescriptions/[^/]+$", + "required": true, + "type": "string" + } + }, + "path": "v1/{+name}", + "response": { + "$ref": "GoogleCloudDatapipelinesV1TransformDescription" + }, + "scopes": [ + "https://www.googleapis.com/auth/cloud-platform" + ] + } + } } } } } } }, - "revision": "20221029", + "revision": "20221106", "rootUrl": "https://datapipelines.googleapis.com/", "schemas": { + "GoogleCloudDatapipelinesV1ArrayValue": { + "description": "Represents an array of values. The elements can be of any type.", + "id": "GoogleCloudDatapipelinesV1ArrayValue", + "properties": { + "elements": { + "description": "The elements of the array.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1FieldValue" + }, + "type": "array" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1AtomicValue": { + "description": "Represents a non-dividable value.", + "id": "GoogleCloudDatapipelinesV1AtomicValue", + "properties": { + "booleanValue": { + "description": "A boolean value.", + "type": "boolean" + }, + "byteValue": { + "description": "An 8-bit signed value.", + "format": "int32", + "type": "integer" + }, + "bytesValue": { + "description": "An array of raw bytes.", + "format": "byte", + "type": "string" + }, + "datetimeValue": { + "$ref": "GoogleTypeDateTime", + "description": "A datetime value." + }, + "decimalValue": { + "$ref": "GoogleTypeDecimal", + "description": "A large decimal value, equivalent to Java BigDecimal." + }, + "doubleValue": { + "description": "A 64-bit floating point value.", + "format": "double", + "type": "number" + }, + "floatValue": { + "description": "A 32-bit floating point value.", + "format": "float", + "type": "number" + }, + "int16Value": { + "description": "A 16-bit signed value.", + "format": "int32", + "type": "integer" + }, + "int32Value": { + "description": "A 32-bit signed value.", + "format": "int32", + "type": "integer" + }, + "int64Value": { + "description": "A 64-bit signed value.", + "format": "int64", + "type": "string" + }, + "stringValue": { + "description": "A string value.", + "type": "string" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1BatchGetTransformDescriptionsResponse": { + "description": "Response message for BatchGetTransformDescriptions", + "id": "GoogleCloudDatapipelinesV1BatchGetTransformDescriptionsResponse", + "properties": { + "transformDescriptions": { + "description": "List of requested transform descriptions.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1TransformDescription" + }, + "type": "array" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1ComputeSchemaRequest": { + "description": "Request message for ComputeSchema", + "id": "GoogleCloudDatapipelinesV1ComputeSchemaRequest", + "properties": { + "config": { + "$ref": "GoogleCloudDatapipelinesV1ConfiguredTransform", + "description": "Required. The configuration for the transform. If this is not a source, then each input with its schema must be set. It is not required to have any outputs set." + }, + "inputSchemas": { + "description": "Optional. In relation to the full pipeline graph, the schemas of the transforms that are used as inputs to the one for `config`. If `config` represents a transform for reading from some resource, then this should be empty. For all other transforms, at least one value must be provided.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1Schema" + }, + "type": "array" + }, + "rawSchema": { + "$ref": "GoogleCloudDatapipelinesV1RawSchemaInfo", + "description": "Optional. If set, this will use the provided raw schema to compute the schema rather than connecting to any resources. Validation will still occur to make sure it is compatible with all input schemas. If the transform is an IO, the IO must support that schema type." + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1ConfiguredTransform": { + "description": "A fully configured transform that can be validated.", + "id": "GoogleCloudDatapipelinesV1ConfiguredTransform", + "properties": { + "config": { + "$ref": "GoogleCloudDatapipelinesV1Row", + "description": "Configuration values provided. These must match the schema provided in the row's schema." + }, + "uniformResourceName": { + "description": "Unique resource name of the transform. This should be the same as the equivalent `TransformDescription` value.", + "type": "string" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1DataflowJobDetails": { "description": "Pipeline job details specific to the Dataflow API. This is encapsulated here to allow for more executors to store their specific details separately.", "id": "GoogleCloudDatapipelinesV1DataflowJobDetails", @@ -400,6 +611,148 @@ }, "type": "object" }, + "GoogleCloudDatapipelinesV1EnumerationValue": { + "description": "Represents a selected value from an EnumerationType.", + "id": "GoogleCloudDatapipelinesV1EnumerationValue", + "properties": { + "name": { + "description": "Name of the enum option.", + "type": "string" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1Field": { + "description": "Info for a single field in the schema.", + "id": "GoogleCloudDatapipelinesV1Field", + "properties": { + "name": { + "description": "Name of the field.", + "type": "string" + }, + "type": { + "$ref": "GoogleCloudDatapipelinesV1FieldType", + "description": "Type info for the field." + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1FieldType": { + "description": "Type info about a field.", + "id": "GoogleCloudDatapipelinesV1FieldType", + "properties": { + "collectionElementType": { + "$ref": "GoogleCloudDatapipelinesV1FieldType", + "description": "If `type` is an array or iterable, this is the type contained in that array or iterable." + }, + "logicalType": { + "$ref": "GoogleCloudDatapipelinesV1LogicalType", + "description": "If `type` is a logical type, this is the info for the specific logical type." + }, + "mapType": { + "$ref": "GoogleCloudDatapipelinesV1MapType", + "description": "If `type` is a map, this is the key and value types for that map." + }, + "nullable": { + "description": "Whether or not this field is nullable.", + "type": "boolean" + }, + "rowSchema": { + "$ref": "GoogleCloudDatapipelinesV1Schema", + "description": "If `type` is a row, this is the schema of that row." + }, + "type": { + "description": "Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set.", + "enum": [ + "TYPE_NAME_UNSPECIFIED", + "TYPE_NAME_BYTE", + "TYPE_NAME_INT16", + "TYPE_NAME_INT32", + "TYPE_NAME_INT64", + "TYPE_NAME_DECIMAL", + "TYPE_NAME_FLOAT", + "TYPE_NAME_DOUBLE", + "TYPE_NAME_STRING", + "TYPE_NAME_DATETIME", + "TYPE_NAME_BOOLEAN", + "TYPE_NAME_BYTES", + "TYPE_NAME_ARRAY", + "TYPE_NAME_ITERABLE", + "TYPE_NAME_MAP", + "TYPE_NAME_ROW", + "TYPE_NAME_LOGICAL_TYPE" + ], + "enumDescriptions": [ + "Type name is not set (generally an error)", + "8-bit signed integer", + "16-bit signed integer", + "32-bit signed integer", + "64-bit signed integer", + "Large decimal type (equivalent to Java BigDecimal)", + "32-bit floating point integer", + "64-bit floating point integer", + "String", + "Datetime", + "Bool", + "Convenience for an ARRAY of BYTE values", + "Array of some other values.", + "Iterable of some other values.", + "Key/Value mapping between values.", + "Struct that follows a particular schema", + "Beam logical type" + ], + "type": "string" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1FieldValue": { + "description": "A single value in a row. The value set must correspond to the correct type from the row's schema.", + "id": "GoogleCloudDatapipelinesV1FieldValue", + "properties": { + "arrayValue": { + "$ref": "GoogleCloudDatapipelinesV1ArrayValue", + "description": "The array value of this field. Corresponds to TYPE_NAME_ARRAY in the schema." + }, + "atomicValue": { + "$ref": "GoogleCloudDatapipelinesV1AtomicValue", + "description": "The atomic value of this field. Must correspond to the correct atomic type in the schema." + }, + "enumValue": { + "$ref": "GoogleCloudDatapipelinesV1EnumerationValue", + "description": "The enum value of this field. Corresponds to TYPE_NAME_LOGICAL_TYPE in the schema if that logical type represents an `EnumerationType` type." + }, + "fixedBytesValue": { + "$ref": "GoogleCloudDatapipelinesV1FixedBytesValue", + "description": "The fixed-length byte collection of this field. Corresponds to TYPE_NAME_LOGICAL_TYPE in the schema if that logical type represents a `FixedBytes` type." + }, + "iterableValue": { + "$ref": "GoogleCloudDatapipelinesV1IterableValue", + "description": "The iterable value of this field. Corresponds to TYPE_NAME_ITERABLE in the schema." + }, + "mapValue": { + "$ref": "GoogleCloudDatapipelinesV1MapValue", + "description": "The map value of this field. Corresponds to TYPE_NAME_MAP in the schema." + }, + "rowValue": { + "$ref": "GoogleCloudDatapipelinesV1Row", + "description": "The row value of this field. Corresponds to TYPE_NAME_ROW in the schema. This row also holds to its own schema." + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1FixedBytesValue": { + "description": "Represents a collection of bytes whose size is the same as the associated FixedBytes size value.", + "id": "GoogleCloudDatapipelinesV1FixedBytesValue", + "properties": { + "value": { + "description": "The raw bytes. It must be exactly the size specified in the schema.", + "format": "byte", + "type": "string" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1FlexTemplateRuntimeEnvironment": { "description": "The environment values to be set at runtime for a Flex Template.", "id": "GoogleCloudDatapipelinesV1FlexTemplateRuntimeEnvironment", @@ -499,6 +852,20 @@ }, "type": "object" }, + "GoogleCloudDatapipelinesV1IterableValue": { + "description": "Represents an iterable of values. The elements can be of any type.", + "id": "GoogleCloudDatapipelinesV1IterableValue", + "properties": { + "elements": { + "description": "The elements of the iterable.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1FieldValue" + }, + "type": "array" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1Job": { "description": "Definition of the job information maintained by the pipeline. Fields in this entity are retrieved from the executor API (e.g. Dataflow API).", "id": "GoogleCloudDatapipelinesV1Job", @@ -718,6 +1085,91 @@ }, "type": "object" }, + "GoogleCloudDatapipelinesV1LogicalType": { + "description": "Represents the input for creating a specified logical type.", + "id": "GoogleCloudDatapipelinesV1LogicalType", + "properties": { + "enumerationType": { + "$ref": "GoogleCloudDatapipelinesV1LogicalTypeEnumerationType", + "description": "The enum represented by this logical type." + }, + "fixedBytes": { + "$ref": "GoogleCloudDatapipelinesV1LogicalTypeFixedBytes", + "description": "The fixed-size byte collection represented by this logical type." + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1LogicalTypeEnumerationType": { + "description": "Represents the Beam EnumerationType logical type.", + "id": "GoogleCloudDatapipelinesV1LogicalTypeEnumerationType", + "properties": { + "values": { + "description": "Names of the values. The numeric value is the same as the index.", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1LogicalTypeFixedBytes": { + "description": "Represents the Beam FixedBytes logical type.", + "id": "GoogleCloudDatapipelinesV1LogicalTypeFixedBytes", + "properties": { + "sizeBytes": { + "description": "Number of bytes to allocate.", + "format": "int32", + "type": "integer" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1MapType": { + "description": "Represents a map in a schema.", + "id": "GoogleCloudDatapipelinesV1MapType", + "properties": { + "mapKeyType": { + "$ref": "GoogleCloudDatapipelinesV1FieldType", + "description": "Key type of the map. Only atomic types are supported." + }, + "mapValueType": { + "$ref": "GoogleCloudDatapipelinesV1FieldType", + "description": "Value type of the map." + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1MapValue": { + "description": "Represents a key/value pairing.", + "id": "GoogleCloudDatapipelinesV1MapValue", + "properties": { + "entries": { + "description": "The entries in the map.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1MapValueEntry" + }, + "type": "array" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1MapValueEntry": { + "description": "A single entry in the map. Each entry must have a unique key.", + "id": "GoogleCloudDatapipelinesV1MapValueEntry", + "properties": { + "key": { + "$ref": "GoogleCloudDatapipelinesV1FieldValue", + "description": "The key value. Only atomic values are supported." + }, + "value": { + "$ref": "GoogleCloudDatapipelinesV1FieldValue", + "description": "The value associated with the key. It may be of any type." + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1Pipeline": { "description": "The main pipeline entity and all the necessary metadata for launching and managing linked jobs.", "id": "GoogleCloudDatapipelinesV1Pipeline", @@ -804,6 +1256,47 @@ }, "type": "object" }, + "GoogleCloudDatapipelinesV1RawSchemaInfo": { + "description": "The raw schema and its type.", + "id": "GoogleCloudDatapipelinesV1RawSchemaInfo", + "properties": { + "rawSchema": { + "description": "The schema.", + "type": "string" + }, + "type": { + "description": "The type of the schema.", + "enum": [ + "RAW_SCHEMA_TYPE_UNSPECIFIED", + "RAW_SCHEMA_TYPE_AVRO" + ], + "enumDescriptions": [ + "The schema type is unknown.", + "The schema is an Avro schema." + ], + "type": "string" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1Row": { + "description": "Represents an Apache Beam row, though the `Any` nature of values is replaced with more concrete representations of valid values.", + "id": "GoogleCloudDatapipelinesV1Row", + "properties": { + "schema": { + "$ref": "GoogleCloudDatapipelinesV1SchemaSource", + "description": "Required. The schema of the row's data." + }, + "values": { + "description": "Required. The values of this Row. A fully built row is required to hold to the schema specified by `schema`.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1FieldValue" + }, + "type": "array" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1RunPipelineRequest": { "description": "Request message for RunPipeline", "id": "GoogleCloudDatapipelinesV1RunPipelineRequest", @@ -931,6 +1424,39 @@ }, "type": "object" }, + "GoogleCloudDatapipelinesV1Schema": { + "description": "Represents a simplified Apache Beam schema.", + "id": "GoogleCloudDatapipelinesV1Schema", + "properties": { + "fields": { + "description": "Fields in the schema. Every field within a schema must have a unique name.", + "items": { + "$ref": "GoogleCloudDatapipelinesV1Field" + }, + "type": "array" + }, + "referenceId": { + "description": "An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository.", + "type": "string" + } + }, + "type": "object" + }, + "GoogleCloudDatapipelinesV1SchemaSource": { + "description": "Holds a schema or a reference to a schema in some repository.", + "id": "GoogleCloudDatapipelinesV1SchemaSource", + "properties": { + "localSchema": { + "$ref": "GoogleCloudDatapipelinesV1Schema", + "description": "Schema located locally with the message." + }, + "referenceId": { + "description": "The `reference_id` value of a schema in a repository.", + "type": "string" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1SdkVersion": { "description": "The version of the SDK used to run the job.", "id": "GoogleCloudDatapipelinesV1SdkVersion", @@ -970,6 +1496,26 @@ "properties": {}, "type": "object" }, + "GoogleCloudDatapipelinesV1TransformDescription": { + "description": "Description of a schema-aware transform, which provides info on how it can be configured.", + "id": "GoogleCloudDatapipelinesV1TransformDescription", + "properties": { + "name": { + "description": "Output only. The full name of this resource formatted as: projects/{project}/locations/{location}/transformDescriptions/{transform_description} `transform_description` is the same as the `uniform_resource_name` field.", + "readOnly": true, + "type": "string" + }, + "options": { + "$ref": "GoogleCloudDatapipelinesV1Schema", + "description": "Available options for configuring the transform." + }, + "uniformResourceName": { + "description": "Unique resource name of the transform.", + "type": "string" + } + }, + "type": "object" + }, "GoogleCloudDatapipelinesV1Workload": { "description": "Workload details for creating the pipeline jobs.", "id": "GoogleCloudDatapipelinesV1Workload", @@ -1017,6 +1563,83 @@ } }, "type": "object" + }, + "GoogleTypeDateTime": { + "description": "Represents civil time (or occasionally physical time). This type can represent a civil time in one of a few possible ways: * When utc_offset is set and time_zone is unset: a civil time on a calendar day with a particular offset from UTC. * When time_zone is set and utc_offset is unset: a civil time on a calendar day in a particular time zone. * When neither time_zone nor utc_offset is set: a civil time on a calendar day in local time. The date is relative to the Proleptic Gregorian Calendar. If year, month, or day are 0, the DateTime is considered not to have a specific year, month, or day respectively. This type may also be used to represent a physical time if all the date and time fields are set and either case of the `time_offset` oneof is set. Consider using `Timestamp` message for physical time instead. If your use case also would like to store the user's timezone, that can be done in another field. This type is more flexible than some applications may want. Make sure to document and validate your application's limitations.", + "id": "GoogleTypeDateTime", + "properties": { + "day": { + "description": "Optional. Day of month. Must be from 1 to 31 and valid for the year and month, or 0 if specifying a datetime without a day.", + "format": "int32", + "type": "integer" + }, + "hours": { + "description": "Optional. Hours of day in 24 hour format. Should be from 0 to 23, defaults to 0 (midnight). An API may choose to allow the value \"24:00:00\" for scenarios like business closing time.", + "format": "int32", + "type": "integer" + }, + "minutes": { + "description": "Optional. Minutes of hour of day. Must be from 0 to 59, defaults to 0.", + "format": "int32", + "type": "integer" + }, + "month": { + "description": "Optional. Month of year. Must be from 1 to 12, or 0 if specifying a datetime without a month.", + "format": "int32", + "type": "integer" + }, + "nanos": { + "description": "Optional. Fractions of seconds in nanoseconds. Must be from 0 to 999,999,999, defaults to 0.", + "format": "int32", + "type": "integer" + }, + "seconds": { + "description": "Optional. Seconds of minutes of the time. Must normally be from 0 to 59, defaults to 0. An API may allow the value 60 if it allows leap-seconds.", + "format": "int32", + "type": "integer" + }, + "timeZone": { + "$ref": "GoogleTypeTimeZone", + "description": "Time zone." + }, + "utcOffset": { + "description": "UTC offset. Must be whole seconds, between -18 hours and +18 hours. For example, a UTC offset of -4:00 would be represented as { seconds: -14400 }.", + "format": "google-duration", + "type": "string" + }, + "year": { + "description": "Optional. Year of date. Must be from 1 to 9999, or 0 if specifying a datetime without a year.", + "format": "int32", + "type": "integer" + } + }, + "type": "object" + }, + "GoogleTypeDecimal": { + "description": "A representation of a decimal value, such as 2.5. Clients may convert values into language-native decimal formats, such as Java's BigDecimal or Python's decimal.Decimal. [BigDecimal]: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/math/BigDecimal.html [decimal.Decimal]: https://docs.python.org/3/library/decimal.html", + "id": "GoogleTypeDecimal", + "properties": { + "value": { + "description": "The decimal value, as a string. The string representation consists of an optional sign, `+` (`U+002B`) or `-` (`U+002D`), followed by a sequence of zero or more decimal digits (\"the integer\"), optionally followed by a fraction, optionally followed by an exponent. An empty string **should** be interpreted as `0`. The fraction consists of a decimal point followed by zero or more decimal digits. The string must contain at least one digit in either the integer or the fraction. The number formed by the sign, the integer and the fraction is referred to as the significand. The exponent consists of the character `e` (`U+0065`) or `E` (`U+0045`) followed by one or more decimal digits. Services **should** normalize decimal values before storing them by: - Removing an explicitly-provided `+` sign (`+2.5` -> `2.5`). - Replacing a zero-length integer value with `0` (`.5` -> `0.5`). - Coercing the exponent character to upper-case, with explicit sign (`2.5e8` -> `2.5E+8`). - Removing an explicitly-provided zero exponent (`2.5E0` -> `2.5`). Services **may** perform additional normalization based on its own needs and the internal decimal implementation selected, such as shifting the decimal point and exponent value together (example: `2.5E-1` <-> `0.25`). Additionally, services **may** preserve trailing zeroes in the fraction to indicate increased precision, but are not required to do so. Note that only the `.` character is supported to divide the integer and the fraction; `,` **should not** be supported regardless of locale. Additionally, thousand separators **should not** be supported. If a service does support them, values **must** be normalized. The ENBF grammar is: DecimalString = '' | [Sign] Significand [Exponent]; Sign = '+' | '-'; Significand = Digits '.' | [Digits] '.' Digits; Exponent = ('e' | 'E') [Sign] Digits; Digits = { '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' }; Services **should** clearly document the range of supported values, the maximum supported precision (total number of digits), and, if applicable, the scale (number of digits after the decimal point), as well as how it behaves when receiving out-of-bounds values. Services **may** choose to accept values passed as input even when the value has a higher precision or scale than the service supports, and **should** round the value to fit the supported scale. Alternatively, the service **may** error with `400 Bad Request` (`INVALID_ARGUMENT` in gRPC) if precision would be lost. Services **should** error with `400 Bad Request` (`INVALID_ARGUMENT` in gRPC) if the service receives a value outside of the supported range.", + "type": "string" + } + }, + "type": "object" + }, + "GoogleTypeTimeZone": { + "description": "Represents a time zone from the [IANA Time Zone Database](https://www.iana.org/time-zones).", + "id": "GoogleTypeTimeZone", + "properties": { + "id": { + "description": "IANA Time Zone Database time zone, e.g. \"America/New_York\".", + "type": "string" + }, + "version": { + "description": "Optional. IANA Time Zone Database version number, e.g. \"2019a\".", + "type": "string" + } + }, + "type": "object" } }, "servicePath": "",get(name, x__xgafv=None)+Gets the transform description associated with the provided uniform resource name. + +Args: + name: string, Required. The full name formatted as "projects/{your-project}/locations/{google-cloud-region}/transformdescriptions/{uniform-resource-name}". (required) + x__xgafv: string, V1 error format. + Allowed values + 1 - v1 error format + 2 - v2 error format + +Returns: + An object of the form: + + { # Description of a schema-aware transform, which provides info on how it can be configured. + "name": "A String", # Output only. The full name of this resource formatted as: projects/{project}/locations/{location}/transformDescriptions/{transform_description} `transform_description` is the same as the `uniform_resource_name` field. + "options": { # Represents a simplified Apache Beam schema. # Available options for configuring the transform. + "fields": [ # Fields in the schema. Every field within a schema must have a unique name. + { # Info for a single field in the schema. + "name": "A String", # Name of the field. + "type": { # Type info about a field. # Type info for the field. + "collectionElementType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # If `type` is an array or iterable, this is the type contained in that array or iterable. + "logicalType": { # Represents the input for creating a specified logical type. # If `type` is a logical type, this is the info for the specific logical type. + "enumerationType": { # Represents the Beam EnumerationType logical type. # The enum represented by this logical type. + "values": [ # Names of the values. The numeric value is the same as the index. + "A String", + ], + }, + "fixedBytes": { # Represents the Beam FixedBytes logical type. # The fixed-size byte collection represented by this logical type. + "sizeBytes": 42, # Number of bytes to allocate. + }, + }, + "mapType": { # Represents a map in a schema. # If `type` is a map, this is the key and value types for that map. + "mapKeyType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Key type of the map. Only atomic types are supported. + "mapValueType": # Object with schema name: GoogleCloudDatapipelinesV1FieldType # Value type of the map. + }, + "nullable": True or False, # Whether or not this field is nullable. + "rowSchema": # Object with schema name: GoogleCloudDatapipelinesV1Schema # If `type` is a row, this is the schema of that row. + "type": "A String", # Specific type of the field. For non-atomic types, the corresponding type info for that non-atomic must be set. + }, + }, + ], + "referenceId": "A String", # An identifier of the schema for looking it up in a repository. This only needs to be set if the schema is stored in a repository. + }, + "uniformResourceName": "A String", # Unique resource name of the transform. +}+