GH-40507: [C++][ORC] Upgrade ORC to 2.0.0 #40508

wgtmac · 2024-03-13T15:06:51Z

Rationale for this change

This PR aims to upgrade to a new major version of Apache ORC: https://orc.apache.org/news/2024/03/08/ORC-2.0.0/

What changes are included in this PR?

This PR upgrades ORC dependency from 1.9.2 to 2.0.0.

Are these changes tested?

Pass the CIs.

Are there any user-facing changes?

No.

GitHub Issue: [C++][ORC] Upgrade ORC to 2.0.0 #40507

wgtmac · 2024-03-13T15:13:54Z

The Apache ORC java library has dropped Java 8 support from v2.0.0. However, I don't think this is an issue here as the orc-core java dependency is only used in the test scope. Am I correct? @lidavidm @jduo

wgtmac · 2024-03-13T15:38:15Z

OK, I'm wrong:

[INFO] Arrow Performance Benchmarks ....................... SKIPPED
[INFO] Arrow Java C Data Interface ........................ SUCCESS [ 57.354 s]
[INFO] Arrow Orc Adapter .................................. FAILURE [ 35.491 s]
[INFO] Arrow Gandiva ...................................... SUCCESS [01:26 min]
[INFO] Arrow Java Dataset ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:03 min (Wall Clock)
[INFO] Finished at: 2024-03-13T15:23:50Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.12.1:testCompile (default-testCompile) on project arrow-orc: Compilation failure
Error:  /arrow/java/adapter/orc/src/test/java/org/apache/arrow/adapter/orc/OrcReaderTest.java:[40,21] error: cannot access OrcFile
Error:    bad class file: /root/.m2/repository/org/apache/orc/orc-core/2.0.0/orc-core-2.0.0.jar(/org/apache/orc/OrcFile.class)
Error:      class file has wrong version 61.0, should be 53.0
Error:      Please remove or make sure it appears in the correct subdirectory of the classpath.
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :arrow-orc

lidavidm · 2024-03-13T16:13:31Z

Hmm, that is test scope, and the other builds pass. So I guess that pipeline uses Java 8 and we could maybe exclude the file?

lidavidm · 2024-03-13T16:14:11Z

That said, I think what we've done is to hold off on package updates requiring Java 8 for now, since we're still supporting Java 8...We were trying to drop Java 8 support but some people spoke out against. Maybe we need to revisit that.

wgtmac · 2024-03-14T01:23:40Z

As orc-core is only used in the test scope, perhaps we can leave it to stay at 1.9.x which is the last major version with java 8 support.

lidavidm · 2024-03-14T01:36:08Z

I added this to the list at #38051

lidavidm · 2024-03-14T01:37:08Z

I think what's going to have to happen is that if someone really needs Java 8 support, they are going to have to maintain the fork or otherwise come up with a strategy, because eventually these CVE fixes pile up and we won't be able to defer the issue any longer

wgtmac · 2024-03-15T00:09:43Z

@github-actions crossbow submit wheel-windows-*

lidavidm · 2024-03-20T13:27:18Z

I think we're going to revisit Java 8 in the next 3-6 months so we can include this PR in that effort.

pitrou · 2024-03-21T16:53:55Z

You can probably split the Java and C++ parts of this PR.

pitrou

@wgtmac Can you please remove the Java change so that this PR focusses on the C++ version bump?

java/adapter/orc/pom.xml

wgtmac · 2024-03-28T01:50:43Z

Just reverted the java module change. Thanks for the reminder! @pitrou

pitrou · 2024-03-28T09:02:56Z

@github-actions crossbow submit -g python -g wheel

pitrou · 2024-03-28T09:03:03Z

@github-actions crossbow submit -g cpp

pitrou · 2024-03-28T09:34:46Z

So it seems this breaks the macOS wheel builds for Python 3.8, 3.9 and 3.10 (but not 3.11 and 3.12).
The error is related to a protobuf symbol, and ORC uses protobuf:

ImportError: dlopen(/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/test-arm64-env/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6google8protobuf2io19ZeroCopyInputStream8ReadCordEPN4absl12lts_202401164CordEi'

The symbol is google::protobuf::io::ZeroCopyInputStream::ReadCord(absl::lts_20240116::Cord*, int).

@wgtmac Is ORC 2.0 building its own protobuf from scratch? Answer: it should not, we're passing PROTOBUF_HOME to ORC's CMake.

pitrou · 2024-03-28T09:38:20Z

Hmm, weirdly, the 3.10 build succeeded when restarted.

pitrou · 2024-03-28T13:30:22Z

@github-actions crossbow submit wheel-macos-*-arm64

pitrou · 2024-03-28T13:44:20Z

Okay, I think I've found it:

ORC build on successful wheel build:

2024-03-28T13:34:25.3459520Z -- PROTOBUF_HOME: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release
2024-03-28T13:34:25.3460120Z -- Could NOT find Protobuf (missing: Protobuf_DIR)
2024-03-28T13:34:25.3460830Z -- Found the Protobuf headers: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release/include
2024-03-28T13:34:25.3461910Z -- Found the Protobuf library: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release/lib/libprotobuf.a
2024-03-28T13:34:25.3463000Z -- Found the Protoc library: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release/lib/libprotoc.a
2024-03-28T13:34:25.3464060Z -- Found the Protoc executable: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx/tools/protobuf/protoc
2024-03-28T13:34:25.3465220Z -- Found the Protobuf static library: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release/lib/libprotobuf.a
2024-03-28T13:34:25.3466370Z -- Found the Protoc static library: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release/lib/libprotoc.a

ORC build on failed wheel build:

2024-03-28T13:33:59.5638710Z -- PROTOBUF_HOME: /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/installed/arm64-osx-static-release
2024-03-28T13:33:59.5643750Z -- Found the Protobuf headers: /opt/homebrew/include
2024-03-28T13:33:59.5644210Z -- Found the Protobuf library: protobuf::libprotobuf
2024-03-28T13:33:59.5644590Z -- Found the Protoc library: protobuf::libprotoc
2024-03-28T13:33:59.5644960Z -- Found the Protoc executable: protobuf::protoc

Note /opt/homebrew/include.

pitrou · 2024-03-28T13:45:33Z

@github-actions crossbow submit wheel-macos-*-arm64

pitrou · 2024-03-28T13:46:34Z

Yuck, this PR is based on an old version of the git repo...

pitrou · 2024-03-28T13:49:21Z

@github-actions crossbow submit wheel-macos-*-arm64

wgtmac · 2024-03-28T14:06:29Z

So it seems this breaks the macOS wheel builds for Python 3.8, 3.9 and 3.10 (but not 3.11 and 3.12). The error is related to a protobuf symbol, and ORC uses protobuf:
ImportError: dlopen(/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/test-arm64-env/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6google8protobuf2io19ZeroCopyInputStream8ReadCordEPN4absl12lts_202401164CordEi'
The symbol is google::protobuf::io::ZeroCopyInputStream::ReadCord(absl::lts_20240116::Cord*, int).

@wgtmac Is ORC 2.0 building its own protobuf from scratch? Answer: it should not, we're passing PROTOBUF_HOME to ORC's CMake.

Right, ORC should use what we provide here:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 4386 to 4396 in a9b2cc2

    
           "-DLZ4_HOME=${ORC_LZ4_ROOT}" 
        
           "-DPROTOBUF_EXECUTABLE=$<TARGET_FILE:${ARROW_PROTOBUF_PROTOC}>" 
        
           "-DPROTOBUF_HOME=${ORC_PROTOBUF_ROOT}" 
        
           "-DPROTOBUF_INCLUDE_DIR=$<TARGET_PROPERTY:${ARROW_PROTOBUF_LIBPROTOBUF},INTERFACE_INCLUDE_DIRECTORIES>" 
        
           "-DPROTOBUF_LIBRARY=$<TARGET_FILE:${ARROW_PROTOBUF_LIBPROTOBUF}>" 
        
           "-DPROTOC_LIBRARY=$<TARGET_FILE:${ARROW_PROTOBUF_LIBPROTOC}>" 
        
           "-DSNAPPY_HOME=${ORC_SNAPPY_ROOT}" 
        
           "-DSNAPPY_INCLUDE_DIR=${ORC_SNAPPY_INCLUDE_DIR}" 
        
           "-DZSTD_HOME=${ORC_ZSTD_ROOT}" 
        
           "-DZSTD_INCLUDE_DIR=$<TARGET_PROPERTY:${ARROW_ZSTD_LIBZSTD},INTERFACE_INCLUDE_DIRECTORIES>" 
        
           "-DZSTD_LIBRARY=$<TARGET_FILE:${ARROW_ZSTD_LIBZSTD}>")

pitrou · 2024-03-28T15:11:04Z

@github-actions crossbow submit -g python -g wheel -g cpp

github-actions · 2024-03-28T15:15:47Z

Revision: 48f6387

Submitted crossbow builds: ursacomputing/crossbow @ actions-33cf00f7e6

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-cpp
test-cuda-python
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-20.04-python-3
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou

CI results look unrelated now, I'll merge

wgtmac · 2024-03-29T01:15:53Z

Thank you for the investigation! @pitrou

conbench-apache-arrow · 2024-03-29T08:07:05Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 4f39e6e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change This PR aims to upgrade to a new major version of Apache ORC: https://orc.apache.org/news/2024/03/08/ORC-2.0.0/ ### What changes are included in this PR? This PR upgrades ORC dependency from 1.9.2 to 2.0.0. ### Are these changes tested? Pass the CIs. ### Are there any user-facing changes? No. * GitHub Issue: apache#40507 Lead-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Gang Wu <ustcwg@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

wgtmac requested a review from lidavidm as a code owner March 13, 2024 15:06

github-actions bot added Component: Java Component: C++ awaiting review Awaiting review labels Mar 13, 2024

lidavidm mentioned this pull request Mar 14, 2024

[Java] Remove Java 8 support in Arrow v18 #38051

Open

7 tasks

This comment was marked as outdated.

Sign in to view

wgtmac mentioned this pull request Mar 19, 2024

[Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) #36026

Closed

wgtmac mentioned this pull request Mar 26, 2024

MINOR: [Java] Bump org.apache.orc:orc-core from 1.9.2 to 2.0.0 in /java #40779

Closed

pitrou requested changes Mar 27, 2024

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 27, 2024

wgtmac commented Mar 28, 2024

View reviewed changes

java/adapter/orc/pom.xml Outdated Show resolved Hide resolved

github-actions bot removed the Component: Java label Mar 28, 2024

This comment was marked as outdated.

Sign in to view

wgtmac and others added 3 commits March 28, 2024 14:47

apacheGH-40507: [C++][ORC] Upgrade ORC to 2.0.0

7809d1c

Update java/adapter/orc/pom.xml

847def9

Debug pb compiler versions

5a5f093

This comment was marked as outdated.

Sign in to view

List homebrew packages

a96c669

pitrou force-pushed the orc_2.0.0 branch from 19adc6f to a96c669 Compare March 28, 2024 13:48

This comment was marked as outdated.

Sign in to view

Remove unused CMake variable

48f6387

pitrou approved these changes Mar 28, 2024

View reviewed changes

pitrou merged commit 4f39e6e into apache:main Mar 28, 2024
33 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Mar 28, 2024

pitrou mentioned this pull request Mar 28, 2024

[C++][ORC] Upgrade ORC to 2.0.0 #40507

Closed

github-actions bot added the awaiting committer review Awaiting committer review label Mar 28, 2024

llama90 mentioned this pull request Apr 22, 2024

[C++] Build failure due to orc_ep in IDE #41331

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40507: [C++][ORC] Upgrade ORC to 2.0.0 #40508

GH-40507: [C++][ORC] Upgrade ORC to 2.0.0 #40508

wgtmac commented Mar 13, 2024 •

edited by github-actions bot

wgtmac commented Mar 13, 2024

wgtmac commented Mar 13, 2024

lidavidm commented Mar 13, 2024

lidavidm commented Mar 13, 2024

wgtmac commented Mar 14, 2024

lidavidm commented Mar 14, 2024

lidavidm commented Mar 14, 2024

wgtmac commented Mar 15, 2024

This comment was marked as outdated.

lidavidm commented Mar 20, 2024

pitrou commented Mar 21, 2024

pitrou left a comment

wgtmac commented Mar 28, 2024

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Mar 28, 2024 •

edited

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

pitrou commented Mar 28, 2024

This comment was marked as outdated.

wgtmac commented Mar 28, 2024

pitrou commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

pitrou left a comment

wgtmac commented Mar 29, 2024

conbench-apache-arrow bot commented Mar 29, 2024

GH-40507: [C++][ORC] Upgrade ORC to 2.0.0 #40508

GH-40507: [C++][ORC] Upgrade ORC to 2.0.0 #40508

Conversation

wgtmac commented Mar 13, 2024 • edited by github-actions bot

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

wgtmac commented Mar 13, 2024

wgtmac commented Mar 13, 2024

lidavidm commented Mar 13, 2024

lidavidm commented Mar 13, 2024

wgtmac commented Mar 14, 2024

lidavidm commented Mar 14, 2024

lidavidm commented Mar 14, 2024

wgtmac commented Mar 15, 2024

This comment was marked as outdated.

lidavidm commented Mar 20, 2024

pitrou commented Mar 21, 2024

pitrou left a comment

Choose a reason for hiding this comment

wgtmac commented Mar 28, 2024

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Mar 28, 2024 • edited

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

pitrou commented Mar 28, 2024

This comment was marked as outdated.

pitrou commented Mar 28, 2024

This comment was marked as outdated.

wgtmac commented Mar 28, 2024

pitrou commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

pitrou left a comment

Choose a reason for hiding this comment

wgtmac commented Mar 29, 2024

conbench-apache-arrow bot commented Mar 29, 2024

wgtmac commented Mar 13, 2024 •

edited by github-actions bot

pitrou commented Mar 28, 2024 •

edited