Skip to content

Beam 2.29.0 Release
Compare
Choose a tag to compare
@emilymye emilymye released this 10 Feb 06:49
· 11159 commits to master since this release
v2.29.0
d7f44ed

NOTE: This version was originally released on 2021-04-29 and added to GitHub releases late.

We are happy to present the new 2.29.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.

For more information on changes in 2.29.0, check out the detailed release notes.

Highlights

  • Spark Classic and Portable runners officially support Spark 3 (BEAM-7093).
  • Official Java 11 support for most runners (Dataflow, Flink, Spark) (BEAM-2530).
  • DataFrame API now supports GroupBy.apply (BEAM-11628).

I/Os

  • Added support for S3 filesystem on AWS SDK V2 (Java) (BEAM-7637)
  • GCP BigQuery sink (file loads) uses runner determined sharding for unbounded data (BEAM-11772)
  • KafkaIO now recognizes the partition property in writing records (BEAM-11806)
  • Support for Hadoop configuration on ParquetIO (BEAM-11913)

New Features / Improvements

Breaking Changes

  • Deterministic coding enforced for GroupByKey and Stateful DoFns. Previously non-deterministic coding was allowed, resulting in keys not properly being grouped in some cases. (BEAM-11719)
    To restore the old behavior, one can register FakeDeterministicFastPrimitivesCoder with
    beam.coders.registry.register_fallback_coder(beam.coders.coders.FakeDeterministicFastPrimitivesCoder())
    or use the allow_non_deterministic_key_coders pipeline option.

Deprecations

  • Support for Flink 1.8 and 1.9 will be removed in the next release (2.30.0) (BEAM-11948).

Known Issues

List of Contributors

According to git shortlog, the following people contributed to the 2.29.0 release. Thank you to all contributors!

Ahmet Altay, Alan Myrvold, Alex Amato, Alexander Chermenin, Alexey Romanenko,
Allen Pradeep Xavier, Amy Wu, Anant Damle, Andreas Bergmeier, Andrei Balici,
Andrew Pilloud, Andy Xu, Ankur Goenka, Bashir Sadjad, Benjamin Gonzalez, Boyuan
Zhang, Brian Hulette, Chamikara Jayalath, Chinmoy Mandayam, Chuck Yang,
dandy10, Daniel Collins, Daniel Oliveira, David Cavazos, David Huntsperger,
David Moravek, Dmytro Kozhevin, Emily Ye, Esun Kim, Evgeniy Belousov, Filip
Popić, Fokko Driesprong, Gris Cuevas, Heejong Lee, Ihor Indyk, Ismaël Mejía,
Jakub-Sadowski, Jan Lukavský, John Edmonds, Juan Sandoval, 谷口恵輔, Kenneth
Jung, Kenneth Knowles, KevinGG, Kiley Sok, Kyle Weaver, MabelYC, Mackenzie
Clark, Masato Nakamura, Milena Bukal, Miltos, Minbo Bae, Miraç Vuslat Başaran,
mynameborat, Nahian-Al Hasan, Nam Bui, Niel Markwick, Niels Basjes, Ning Kang,
Nir Gazit, Pablo Estrada, Ramazan Yapparov, Raphael Sanamyan, Reuven Lax, Rion
Williams, Robert Bradshaw, Robert Burke, Rui Wang, Sam Rohde, Sam Whittle,
Shehzaad Nakhoda, Shehzaad Nakhoda, Siyuan Chen, Sonam Ramchand, Steve Niemitz,
sychen, Sylvain Veyrié, Tim Robertson, Tobias Kaymak, Tomasz Szerszeń, Tomasz
Szerszeń, Tomo Suzuki, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Yichi
Zhang, Yifan Mai, Yixing Zhang, Yoshiki Obata

Assets 2