Releases · apache/beam

04 Mar 19:21

v2.37.0

d97c521

Beam 2.37.0 release

We are happy to present the new 2.37.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.37.0 check out the detailed release notes.

Highlights

Java 17 support for Dataflow (BEAM-12240).
- Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (BEAM-13695).
- ZetaSql is currently unsupported (issue).
Python 3.9 support in Apache Beam (BEAM-12000).
- Dataflow support for Python 3.9 is expected to be available with 2.37.0,
  but may not be fully available yet when the release is announced (BEAM-13864).
- Users of Dataflow Runner V2 can run Python 3.9 pipelines with 2.37.0 release right away.

I/Os

Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each.
- JDBCIO (BEAM-13293).
- Debezium (BEAM-13761).
- BeamSQL (BEAM-13683).
- BiqQuery (BEAM-13732).
- KafkaIO now also has automatic expansion service startup. (BEAM-13821).

New Features / Improvements

DataFrame API now supports pandas 1.4.x (BEAM-13605).
Go SDK DoFns can now observe trigger panes directly (BEAM-13757).

Known Issues

See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.37.0 release. Thank you to all contributors!

Aizhamal Nurmamat kyzy
Alexander
Alexander Chermenin
Alexandr Zhuravlev
Alexey Romanenko
Anand Inguva
andoni-guzman
andreukus
Andy Ye
Artur Khanin
Aydar Farrakhov
Aydar Zainutdinov
AydarZaynutdinov
Benjamin Gonzalez
Brian Hulette
Chamikara Jayalath
Daniel Oliveira
Danny McCormick
daria-malkova
daria.malkova
darshan-sj
David Huntsperger
dprieto91
emily
Etienne Chauchot
Fernando Morales
Heejong Lee
Ismaël Mejía
Jack McCluskey
Jan Lukavský
johnjcasey
Kamil Breguła
kellen
Kenneth Knowles
kileys
Kyle Weaver
Luke Cwik
Marcin Kuthan
Marco Robles
Matt Rudary
Miguel Hernandez
Milena Bukal
Moritz Mack
Mostafa Aghajani
Ning Kang
Pablo Estrada
Pavel Avilov
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Sam Whittle
Sandy Chapman
Sergey Kalinin
Thiago Nunes
thorbjorn444
Tim Robertson
Tomo Suzuki
Valentyn Tymofieiev
Victor
Victor Chen
Vitaly Ivanov
Yichi Zhang

Assets 2

08 Feb 00:08

emilymye

v2.36.0

81f3a16

Beam 2.36.0 release

We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.36.0, check out the detailed release
notes.

I/Os

Support for stopReadTime on KafkaIO SDF (Java).(BEAM-13171).

New Features / Improvements

Added support for cloudpickle as a pickling library for Python SDK (BEAM-8123). To use cloudpickle, set pipeline option: --pickler_lib=cloudpickle
Added option to specify triggering frequency when streaming to BigQuery (Python) (BEAM-12865).
Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (BEAM-13459). To enable, set pipeline option: --enable_artifact_caching, this will be enabled by default in a future release.

Breaking Changes

Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to this page to update it. (BEAM-12092).
Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public BEAM-13638.
Properly check output timestamps on elements output from DoFns, timers, and onWindowExpiration in Java BEAM-12931.
Fixed a bug with DeferredDataFrame.xs when used with a non-tuple key
(BEAM-13421).

Known Issues

Users may encounter an unexpected java.lang.ArithmeticException when outputting a timestamp
for an element further than allowedSkew from an allowed DoFN skew set to a value more than
Integer.MAX_VALUE.
See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.36.0 release. Thank you to all contributors!

Ada Wong
Ahmet Altay
Alexander
Alexander Dahl
Alexandr Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
Andrew Pilloud
Andy Ye
Arkadiusz Gasiński
Artur Khanin
Arun Pandian
Aydar Farrakhov
Aydar Zainutdinov
AydarZaynutdinov
Benjamin Gonzalez
Brian Hulette
Chamikara Jayalath
Daniel Collins
Daniel Oliveira
Daniel Thevessen
Daniela Martín
David Hinkes
David Huntsperger
Emily Ye
Etienne Chauchot
Evan Galpin
Heejong Lee
Ilya
Ilya Kozyrev
In-Ho Yi
Jack McCluskey
Janek Bevendorff
Jarek Potiuk
Ke Wu
KevinGG
Kyle Hersey
Kyle Weaver
Luís Bianchin
Luke Cwik
Masato Nakamura
Matthias Baetens
Mehdi Drissi
Melissa Pashniak
Michel Davit
Miguel Hernandez
MiguelAnzoWizeline
Milena Bukal
Moritz Mack
Mostafa Aghajani
Nathan J Mehl
Niel Markwick
Ning Kang
Pablo Estrada
Pavel Avilov
Quentin Sommer
Reuben van Ammers
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ryan Thompson
Sam Whittle
Sayat
Sergei Lebedev
Sergey Kalinin
Steve Niemitz
Talat Uyarer
Thiago Nunes
Tianyang Hu
Tim Robertson
Valentyn Tymofieiev
Vitaly Ivanov
Yichi Zhang
Yiru Tang
Yu Feng
Yu ISHIKAWA
Zachary Houfek
blais
daria-malkova
daria.malkova
darshan-sj
dpcollins-google
emily
ewianda
johnjcasey
kileys
lam206
laraschmidt
mosche
msbukal@google.com
tvalentyn

Assets 2

30 Dec 02:00

tvalentyn

v2.35.0

a3400e3

Beam 2.35.0 release

We are happy to present the new 2.35.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.35.0, check out the detailed release
notes.

Highlights

MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
Upgrades Log4j version used in test suites (Apache Beam testing environment only, not for end user consumption) to 2.17.0(BEAM-13434).
Note that Apache Beam versions do not depend on the Log4j 2 dependency (log4j-core) impacted by CVE-2021-44228.
However we urge users to update direct and indirect dependencies (if any) on Log4j 2 to the latest version by updating their build configuration and redeploying impacted pipelines.

I/Os

We changed the data type for ranges in JdbcIO.readWithPartitions from int to long (BEAM-13149).
This is a relatively minor breaking change, which we're implementing to improve the usability of the transform without increasing cruft.
This transform is relatively new, so we may implement other breaking changes in the future to improve its usability.
Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).

New Features / Improvements

Added custom delimiters to Python TextIO reads (BEAM-12730).
Added escapechar parameter to Python TextIO reads (BEAM-13189).
Splittable reading is enabled by default while reading data with ParquetIO (BEAM-12070).
DoFn Execution Time metrics added to Go (BEAM-13001).
Cross-bundle side input caching is now available in the Go SDK for runners that support the feature by setting the EnableSideInputCache hook (BEAM-11097).
Upgraded the GCP Libraries BOM version to 24.0.0 and associated dependencies (BEAM-11205). For Google Cloud client library versions set by this BOM,
see this table.
Removed avro-python3 dependency in AvroIO. Fastavro has already been our Avro library of choice on Python 3. Boolean use_fastavro is left for api compatibility, but will have no effect.(BEAM-13016).
MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
Remote packages can now be downloaded from locations supported by apache_beam.io.filesystems. The files will be downloaded on Stager and uploaded to staging location. For more information, see BEAM-11275

Breaking Changes

A new URN convention was adopted for cross-language transforms and existing URNs were updated. This may break advanced use-cases, for example, if a custom expansion service is used to connect diffrent Beam Java and Python versions. (BEAM-12047).
The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING function in SqlTransform, when used with the Calcite dialect (BEAM-13099, CALCITE-4427).
ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-13233).

Deprecations

Non-splittable reading is deprecated while reading data with ParquetIO (BEAM-12070).

Bugfixes

Properly map main input windows to side input windows by default (Go)
(BEAM-11087).
Fixed data loss when writing to DynamoDB without setting deduplication key names (Java)
(BEAM-13009).
Go SDK Examples now have types and functions registered. (Go) (BEAM-5378)

Known Issues

Users of beam-sdks-java-io-hcatalog (and beam-sdks-java-extensions-sql-hcatalog) must take care to override the transitive log4j dependency when they add a hive dependency (BEAM-13499).

List of Contributors

According to git shortlog, the following people contributed to the 2.35.0 release. Thank you to all contributors!

Ahmet Altay
Alexandr Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
Andrew Pilloud
Ankur Goenka
Anthony Sottile
Artur Khanin
Aydar Farrakhov
Aydar Zainutdinov
Benjamin Gonzalez
brachipa
Brian Hulette
Calvin Leung
Chamikara Jayalath
Chris Gray
Damon Douglas
Daniel Collins
Daniel Oliveira
daria.malkova
darshan-sj
David Huntsperger
Dmitrii Kuzin
dpcollins-google
dprieto
egalpin
Etienne Chauchot
Eugene Nikolaiev
Fernando Morales
Hector Lagos
Heejong Lee
Ilya Kozyrev
Iñigo San Jose Visiers
Jack McCluskey
Jiayang Wu
jrhy
Kenneth Knowles
KevinGG
kileys
klmilam
Kyle Weaver
Luís Bianchin
Luke Cwik
Melissa Pashniak
Michael Luckey
Miguel Hernandez
Milena Bukal
Minbo Bae
minherz
Moritz Mack
mosche
Natalie
Ning Kang
Pablo Estrada
Pavel Avilov
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Rogan Morrow
Ruslan Altynnikov
Sam Whittle
Sergey Kalinin
Slava Chernyak
Svetak Sundhar
Tianyang Hu
Tim Robertson
Tomo Suzuki
tuorhador
Udi Meiri
vachan-shetty
Valentyn Tymofieiev
Yichi Zhang
zhoufek

Assets 2

11 Nov 20:03

ibzib

v2.34.0

b3b1843

Beam 2.34.0 release

We are happy to present the new 2.34.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.34.0, check out the detailed release
notes.

Highlights

The Beam Java API for Calcite SqlTransform is no longer experimental (BEAM-12680).
Python's ParDo (Map, FlatMap, etc.) transforms now suport a with_exception_handling option for easily ignoring bad records and implementing the dead letter pattern.

I/Os

ReadFromBigQuery and ReadAllFromBigQuery now run queries with BATCH priority by default. The query_priority parameter is introduced to the same transforms to allow configuring the query priority (Python) (BEAM-12913).
[EXPERIMENTAL] Support for BigQuery Storage Read API added to ReadFromBigQuery. The newly introduced method parameter can be set as DIRECT_READ to use the Storage Read API. The default is EXPORT which invokes a BigQuery export request. (Python) (BEAM-10917).
[EXPERIMENTAL] Added use_native_datetime parameter to ReadFromBigQuery to configure the return type of DATETIME fields when using ReadFromBigQuery. This parameter can only be used when method = DIRECT_READ(Python) (BEAM-10917).

New Features / Improvements

Upgrade to Calcite 1.26.0 (BEAM-9379).
Added a new dataframe extra to the Python SDK that tracks pandas versions
we've verified compatibility with. We now recommend installing Beam with pip install apache-beam[dataframe] when you intend to use the DataFrame API
(BEAM-12906).
Add an example of deploying Python Apache Beam job with Spark Cluster

Breaking Changes

SQL Rows are no longer flattened (BEAM-5505).
[Go SDK] beam.TryCrossLanguage's signature now matches beam.CrossLanguage. Like other Try functions it returns an error instead of panicking. (BEAM-9918).
BEAM-12925 was fixed. It used to silently pass incorrect null data read from JdbcIO. Pipelines affected by this will now start throwing failures instead of silently passing incorrect data.

Bugfixes

Fixed error while writing multiple DeferredFrames to csv (Python) (BEAM-12701).
Fixed error when importing the DataFrame API with pandas 1.0.x installed (BEAM-12945).
Fixed top.SmallestPerKey implementation in the Go SDK (BEAM-12946).

List of Contributors

According to git shortlog, the following people contributed to the 2.34.0 release. Thank you to all contributors!

Ahmet Altay,
Aizhamal Nurmamat kyzy,
Alex Amato,
Alexander Chermenin,
Alexey Romanenko,
AlikRodriguez,
Andrew Pilloud,
Andy Xu,
Ankur Goenka,
Aydar Farrakhov,
Aydar Zainutdinov,
Aydar Zaynutdinov,
AydarZaynutdinov,
Benjamin Gonzalez,
BenWhitehead,
Brachi Packter,
Brian Hulette,
Bu Sun Kim,
Chamikara Jayalath,
Chris Gray,
Chuck Yang,
Chun Yang,
Claire McGinty,
comet,
Daniel Collins,
Daniel Oliveira,
Daniel Thevessen,
daria.malkova,
David Cavazos,
David Huntsperger,
Dmytro Kozhevin,
dpcollins-google,
Eduardo Sánchez López,
Elias Djurfeldt,
emily,
Emily Ye,
Enis Sert,
Etienne Chauchot,
Fernando Morales,
Heejong Lee,
Ihor Indyk,
Ismaël Mejía,
Israel Herraiz,
Jack McCluskey,
Jonathan Hourany,
Judah Rand,
Kenneth Knowles,
KevinGG,
Ke Wu,
kileys,
Kyle Weaver,
Luke Cwik,
masahitojp,
MiguelAnzoWizeline,
Minbo Bae,
Niels Basjes,
Ning Kang,
Pablo Estrada,
pareshsarafmdb,
Paul Féraud,
Piotr Szczepanik,
Reuven Lax,
Ritesh Ghorse,
R. Miles McCain,
Robert Bradshaw,
Robert Burke,
Rogan Morrow,
Ruwan Lambrichts,
rvballada,
Ryan Thompson,
Sam Rohde,
Sam Whittle,
Ștefan Istrate,
Steve Niemitz,
Thomas Li Fredriksen,
Tomo Suzuki,
tvalentyn,
Udi Meiri,
Vachan,
Valentyn Tymofieiev,
Vincent Marquez,
WinsonT,
Yichi Zhang,
Yifan Mai,
Yilei "Dolee" Yang,
zhoufek

Assets 2

12 Oct 16:53

udim

v2.33.0

ae8431f

Beam 2.33.0 release

We are happy to present the new 2.33.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.33.0, check out the detailed release
notes.

Highlights

Go SDK is no longer experimental, and is officially part of the Beam release process.
- Matching Go SDK containers are published on release.
- Batch usage is well supported, and tested on Flink, Spark, and the Python Portable Runner.
  - SDK Tests are also run against Google Cloud Dataflow, but this doesn't indicate reciprocal support.
- The SDK supports Splittable DoFns, Cross Language transforms, and most Beam Model basics.
- Go Modules are now used for dependency management.
  - This is a breaking change, see Breaking Changes for resolution.
  - Easier path to contribute to the Go SDK, no need to set up a GO_PATH.
  - Minimum Go version is now Go v1.16
- See the announcement blogpost for full information once published.

New Features / Improvements

Projection pushdown in SchemaIO (BEAM-12609).
Upgrade Flink runner to Flink versions 1.13.2, 1.12.5 and 1.11.4 (BEAM-10955).

Breaking Changes

Since release 2.30.0, "The AvroCoder changes for BEAM-2303 [changed] the reader/writer from the Avro ReflectDatum* classes to the SpecificDatum* classes" (Java). This default behavior change has been reverted in this release. Use the useReflectApi setting to control it (BEAM-12628).

Deprecations

Python GBK will stop supporting unbounded PCollections that have global windowing and a default trigger in Beam 2.34. This can be overriden with --allow_unsafe_triggers. (BEAM-9487).
Python GBK will start requiring safe triggers or the --allow_unsafe_triggers flag starting with Beam 2.34. (BEAM-9487).