Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NoClassDefFoundError for Apache Arrow class when using JDK 17 #2060

Closed
mpeddada1 opened this issue May 23, 2022 · 2 comments · Fixed by #2165
Closed

NoClassDefFoundError for Apache Arrow class when using JDK 17 #2060

mpeddada1 opened this issue May 23, 2022 · 2 comments · Fixed by #2165
Assignees
@prash-mi @mpeddada1
Labels
api: bigquery Issues related to the googleapis/java-bigquery API. external This issue is blocked on a bug with the actual product. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@mpeddada1
Copy link
Contributor

mpeddada1 commented May 23, 2022 •
edited

Running tests in Java 17 is resulting in some failures with Apache Arrow at the moment. A couple of tests are failing with the following log output:

[INFO] Running com.google.cloud.bigquery.it.ITBigQueryTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
May 23, 2022 10:13:47 AM com.google.cloud.bigquery.ConnectionImpl highThroughPutRead
INFO: 
 Using BigQuery Read API
Exception in thread "pool-16-thread-1" java.lang.ExceptionInInitializerError
	at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228)
	at org.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223)
	at org.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.processRows(ConnectionImpl.java:913)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.access$300(ConnectionImpl.java:879)
	at com.google.cloud.bigquery.ConnectionImpl.lambda$processArrowStreamAsync$6(ConnectionImpl.java:855)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Failed to initialize MemoryUtil.
	at org.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:136)
	... 11 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @36aa7bc2
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at org.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:84)
	... 11 more
May 23, 2022 10:13:55 AM com.google.cloud.bigquery.ConnectionImpl highThroughPutRead
INFO: 
 Using BigQuery Read API
Exception in thread "pool-20-thread-1" java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.memory.util.MemoryUtil
	at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228)
	at org.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223)
	at org.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.processRows(ConnectionImpl.java:913)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.access$300(ConnectionImpl.java:879)
	at com.google.cloud.bigquery.ConnectionImpl.lambda$processArrowStreamAsync$6(ConnectionImpl.java:855)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
May 23, 2022 10:14:51 AM com.google.cloud.bigquery.ConnectionImpl lambda$populateBufferAsync$3
WARNING: 
pool-24-thread-3 Interrupted
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1638)
	at java.base/java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:485)
	at java.base/java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:673)
	at com.google.cloud.bigquery.ConnectionImpl.lambda$populateBufferAsync$3(ConnectionImpl.java:548)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

May 23, 2022 10:14:51 AM com.google.cloud.bigquery.ConnectionImpl lambda$parseRpcDataAsync$2
WARNING: 
pool-24-thread-2 Interrupted @ parseRpcDataAsync
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1638)
	at java.base/java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:485)
	at java.base/java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:673)
	at com.google.cloud.bigquery.ConnectionImpl.lambda$parseRpcDataAsync$2(ConnectionImpl.java:507)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

May 23, 2022 10:14:53 AM com.google.cloud.bigquery.ConnectionImpl lambda$runNextPageTaskAsync$1
WARNING: 
pool-24-thread-1 Interrupted @ runNextPageTaskAsync
May 23, 2022 10:16:53 AM com.google.cloud.bigquery.ConnectionImpl highThroughPutRead
INFO: 
 Using BigQuery Read API
Exception in thread "pool-26-thread-1" java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.memory.util.MemoryUtil
	at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228)
	at org.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223)
	at org.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.processRows(ConnectionImpl.java:913)
	at com.google.cloud.bigquery.ConnectionImpl$ArrowRowReader.access$300(ConnectionImpl.java:879)
	at com.google.cloud.bigquery.ConnectionImpl.lambda$processArrowStreamAsync$6(ConnectionImpl.java:855)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
[ERROR] Tests run: 111, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 440.305 s <<< FAILURE! - in com.google.cloud.bigquery.it.ITBigQueryTest
[ERROR] com.google.cloud.bigquery.it.ITBigQueryTest.testBQResultSetPaginationSlowQuery  Time elapsed: 5.784 s  <<< FAILURE!
java.lang.AssertionError: expected:<300000> but was:<0>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at com.google.cloud.bigquery.it.ITBigQueryTest.testBQResultSetPaginationSlowQuery(ITBigQueryTest.java:2575)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:833)

List of failing tests:

  1. ITBigQueryTest.testBQResultSetPaginationSlowQuery
  2. ITBigQueryTest.testReadAPIConnectionMultiClose
  3. ITBigQueryTest.testReadAPIIterationAndOrder
  4. ITNightlyBigQueryTest.testIterateAndOrder

Steps to reproduce:

  1. sdk use java 17.0.3-oracle
  2. java -version
java version "17.0.3" 2022-04-19 LTS
Java(TM) SE Runtime Environment (build 17.0.3+8-LTS-111)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.3+8-LTS-111, mixed mode, sharing)
  1. mvn verify -Penable-integration-tests

This issue looks related: https://issues.apache.org/jira/browse/ARROW-16472

cc @stephaniewang526, @prash-mi
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/java-bigquery API. label May 23, 2022
@meredithslota meredithslota added external This issue is blocked on a bug with the actual product. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels May 24, 2022
@mpeddada1 mpeddada1 mentioned this issue May 24, 2022
Merged
@mpeddada1
Copy link
Contributor Author

Thanks for offering to help with this, @prash-mi! I'm curious to know if you are also able to see this in standard Java 17 in your environment?

@prash-mi prash-mi mentioned this issue Jul 21, 2022
Closed
@mpeddada1 mpeddada1 mentioned this issue Jul 22, 2022
Merged
@mpeddada1
Copy link
Contributor Author

Fixed with #2165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prash-mi prash-mi

@mpeddada1 mpeddada1

Labels
api: bigquery Issues related to the googleapis/java-bigquery API. external This issue is blocked on a bug with the actual product. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(java): make field accessible to address Java 17 issue with arrow googleapis/java-bigquery
3 participants
@meredithslota @prash-mi @mpeddada1