sync with open source how #118
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add DataflowRunner in wordCount run command with Gradle add DataflowRunner in wordCount run command with Gradle * Update dataflow.md to update mvn package and run commands Update dataflow.md to update mvn package and run commands
* replace Periodic to Unbounded for BigTableIOST * replace PeriodicImpulse for Kafka stress test * refactor * correct a number of records and records per second * specify correct resources for kafka servers * refactor
* Stabilize additional teststream cases. * Update sdks/go/test/integration/primitives/teststream_test.go Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> * Update sdks/go/test/integration/primitives/teststream.go Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> * Update sdks/go/test/integration/primitives/teststream_test.go Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com>
…ge supplied in --setup_file option. (#31070)
…1050) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#31051) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.22.0 to 0.24.0. - [Commits](golang/net@v0.22.0...v0.24.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…anslation (#30910) * iceberg write schematransform and test * IcebergIO translation and tests * unify iceberg urns and identifiers; update some comments * replace icebergIO translation with iceberg schematransform translation; fix Schema::sorted to do recursive sorting * removed new proto file and moved Managed URNs to beam_runner_api.proto; we now use SchemaTransformPayload for all schematransforms, including Managed; adding a version number to FileWriteResult encoding so that we can use it to fork in the future whhen needed * Row and Schema snake_case <-> camelCase conversion logic * Row sorted() util * use Row::sorted to fetch Managed & Iceberg row configs * use snake_case convention when translating transforms to spec; remove Managed and Iceberg urns from proto and use SCHEMA_TRANSFORM URN * perform snake_case <-> camelCase conversions directly in TypedSchemaTransformProvider * add SchemaTransformTranslation abstraction. when encountering a SCHEMA_TRANSFORM urn, fetch underlying identifier * prioritize registered providers; remove snake_case <-> camelCase conversions from python side
* [YAML] - Normalize YAML PubSub format * [YAML] - Fix format * [YAML] - Fix import * [YAML] - Update tests and web
Using `View#asList()` can be *significantly* more expesnive than `View#asIterable()` as the former creates a distribtued mapping of range(0, N) to the elements, and iterating over it (while respecting this order) requries O(N) non-continguous point lookups. These counters should be useful in diagnosing performance issues.
* Fix building release candidate * String change
Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>
…; use manifest files
#30770) * fix several bugs regarding avto <-> beam schema conversion * fix test with my bizzare workaround * fix import in test * fix linting issue * comments -> docstrings * formatting * review comments * add skip test config to sqltransform test * fix docstrings * formatting * review comment * uncomment test skip * always return map in beam_type_to_avro_type * unnest primitive types for arrays and maps too * remove print stmt
…ly in element batches. (#31319) * Don't swallow data errors on timer errors. * reset error in case of timer handling * Don't drop timers on splits. --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>
* BigQueryIO write throttling detection * double retry interval
* Logging loading filesystems failures. --------- Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
Bumps com.gradle.enterprise from 3.17.2 to 3.17.4. --- updated-dependencies: - dependency-name: com.gradle.enterprise dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#31344) * Disable soft delete policy when creating new default bucket. * Fix lints * Add a check on returned bucket name.
… enabled (#31358) * Add warning to gcpTempLocation when its bucket has soft delete policy enabled. * Fix an issue reported by spotbugs.
* Don't set SETUPTOOLS_USE_DISTUTILS=stdlib since it doesn't work on Py3.12 * Unblock tests.
…hentication (#31352) * Updates the Java Expansion Service container to support Google ALTS authentication * Fix spotless * Updates the description
This reverts commit 557b5ba.
Fix beam quest link
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.