Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

Closed
suztomo opened this issue Jul 20, 2021 · 6 comments
Closed

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

suztomo opened this issue Jul 20, 2021 · 6 comments
Labels
p3

Comments

@suztomo
Copy link
Contributor

suztomo commented Jul 20, 2021 •
edited

googleapis/java-bigquery#1374 (comment)

Jul 20, 2021 6:34:47 PM com.google.cloud.tools.dependencies.linkagemonitor.LinkageMonitor run
SEVERE: Newly introduced problems:
Class org.joda.time.LocalDate is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Days is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime$Property is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.DateTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateTimeZone is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.DateTimeFormatter is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.ISODateTimeFormat is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadablePeriod is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Period is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableInstant is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDate$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)

org.apache.avro:avro:1.8.2 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.apache.avro:avro:1.8.2 (compile)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.codehaus.jackson:jackson-mapper-asl:1.9.13 (compile)

https://github.com/googleapis/java-bigquery/pull/1374/checks?check_run_id=3117079174
@suztomo suztomo mentioned this issue Jul 20, 2021
Merged
@suztomo
Copy link
Contributor Author

suztomo commented Jul 21, 2021 •
edited

This gives the same errors.

  @Test
  public void testAvroDependency() throws Exception {
    Artifact avro =
        new DefaultArtifact("org.apache.avro:avro:1.8.2");
    ClassPathResult classPathResult =
        new ClassPathBuilder()
            .resolve(ImmutableList.of(avro), false, DependencyMediation.MAVEN);

    System.out.println(classPathResult);

    LinkageChecker linkageChecker = LinkageChecker.create(classPathResult.getClassPath());

    ImmutableSet<LinkageProblem> linkageProblems = linkageChecker.findLinkageProblems();
    Truth.assertThat(linkageProblems).isEmpty();
  }

Doesn't avro declare joda-time dependency? It's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Linkage Monitor sets full=true to traverse optional dependencies. Why joda-time not included in the list?

    ClassPathResult classPathResult =
        (new ClassPathBuilder())
            .resolve(snapshotManagedDependencies, true, DependencyMediation.MAVEN);

In fact, flipping the flag from false to true in the unit test fixes the error.

Investigate why Linkage Monitor doesn't add joda-time there.

@suztomo
Copy link
Contributor Author

suztomo commented Jul 21, 2021

@stephaniewang526 It turned out joda-time is explicitly excluded in the pom.xml (after the flatten plugin; see below). Is this expected?

suztomo-macbookpro44% vim ~/.m2/repository/com/google/cloud/google-cloud-bigquery/1.135.4-SNAPSHOT/google-cloud-bigquery-1.135.4-SNAPSHOT.pom
...
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.8.2</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>paranamer</artifactId>
          <groupId>com.thoughtworks.paranamer</groupId>
        </exclusion>
        <exclusion>
          <artifactId>snappy-java</artifactId>
          <groupId>org.xerial.snappy</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>xz</artifactId>
          <groupId>org.tukaani</groupId>
        </exclusion>
        <exclusion>
          <artifactId>joda-time</artifactId>
          <groupId>joda-time</groupId>
        </exclusion>
        <exclusion>

@suztomo suztomo changed the title Investigate Avro linkage errors. If they are safe, add them to suppress list Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list Jul 21, 2021
@suztomo
Copy link
Contributor Author

suztomo commented Jul 21, 2021 •
edited

Dependency tree of google-cloud-bigquery (commit 02cfa971b805a49a17c497e96d624283b8775ccc):

suztomo-macbookpro44% mvn dependency:tree
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] ---------------< com.google.cloud:google-cloud-bigquery >---------------
[INFO] Building BigQuery 1.135.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ google-cloud-bigquery ---
[INFO] com.google.cloud:google-cloud-bigquery:jar:1.135.4-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-core:jar:1.95.4:compile
[INFO] |  +- com.google.protobuf:protobuf-java-util:jar:3.17.3:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.14:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:0.26.0:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
[INFO] |  \- com.google.protobuf:protobuf-java:jar:3.17.3:compile
[INFO] +- com.google.cloud:google-cloud-core-http:jar:1.95.4:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.32.1:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
[INFO] |  +- com.google.api:gax-httpjson:jar:0.83.0:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.28.0:compile
[INFO] |  \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
[INFO] +- com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile
[INFO] +- com.google.cloud:google-cloud-datacatalog:jar:1.4.2:test
[INFO] |  +- io.grpc:grpc-api:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-context:jar:1.39.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.39.0:test
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1:jar:0.41.2:test
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:test
[INFO] |  +- commons-codec:commons-codec:jar:1.15:test
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.7:compile
[INFO] |  +- com.google.api:gax-grpc:jar:1.66.0:test
[INFO] |  +- io.grpc:grpc-auth:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.39.0:test
[INFO] |  +- io.perfmark:perfmark-api:jar:0.23.0:test
[INFO] |  +- io.grpc:grpc-core:jar:1.39.0:test
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:test
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:test
[INFO] |  +- io.grpc:grpc-alts:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.39.0:test
[INFO] |  \- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:test
[INFO] +- com.google.api.grpc:proto-google-cloud-datacatalog-v1:jar:1.4.2:test
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.117.1:test
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.31.5:compile
[INFO] |  +- com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
[INFO] |  \- com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.5:test
[INFO] +- com.google.auto.value:auto-value-annotations:jar:1.8.1:provided
[INFO] +- com.google.http-client:google-http-client:jar:1.39.2:compile
[INFO] +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
[INFO] +- com.google.auth:google-auth-library-oauth2-http:jar:0.26.0:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- com.google.apis:google-api-services-bigquery:jar:v2-rev20210617-1.32.1:compile
[INFO] +- com.google.cloud:google-cloud-core:jar:tests:1.95.4:test
[INFO] +- com.google.api:api-common:jar:1.10.4:compile
[INFO] +- com.google.guava:guava:jar:30.1.1-android:compile
[INFO] +- com.google.api:gax:jar:1.66.0:compile
[INFO] +- org.threeten:threetenbp:jar:1.5.1:compile
[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] +- com.google.errorprone:error_prone_annotations:jar:2.7.1:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.google.truth:truth:jar:1.1.3:test
[INFO] |  +- org.checkerframework:checker-qual:jar:3.13.0:test
[INFO] |  \- org.ow2.asm:asm:jar:9.1:test
[INFO] +- org.mockito:mockito-core:jar:2.28.2:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.9.10:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.9.10:test
[INFO] |  \- org.objenesis:objenesis:jar:2.6:test
[INFO] \- org.assertj:assertj-core:jar:2.9.1:test

mvn dependency:tree shows joda-time is not there.

[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile

https://search.maven.org/artifact/org.apache.avro/avro/1.8.2/bundle shows it's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Maybe the flutten plugin somehow excluded joda-time because it's optional

@meltsufin meltsufin added the p3 label Jul 21, 2021
@suztomo
Copy link
Contributor Author

suztomo commented Jul 30, 2021

Stephanie was thinking whether Apache Avro or Apache Arrow is better choice. Not taking action for now.

@stephaniewang526
Copy link

We are no longer using Avro. We are using Arrow instead. Hence, we can close this ticket.

@suztomo
Copy link
Contributor Author

suztomo commented May 2, 2022

Thank you for update.

@suztomo suztomo closed this as completed May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
p3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@suztomo @stephaniewang526 @meltsufin