Null pointer exception when running akka-grpc related tests

[akkie]

Versions used

Akka gRPC: 0.8.4
Play gRPC: 0.8.4
Akka version: 2.6.5

Expected Behavior

Running tests should not end in a null pointer exception

Actual Behavior

This is a really strange behavior. If I run multiple tests with sbt test I get the following exception:

 CAUSED BY
[error]  java.lang.NullPointerException: null (HttpEntity.scala:334)
[error] akka.http.scaladsl.model.HttpEntity$Strict.isKnownEmpty(HttpEntity.scala:334)
[error] akka.http.scaladsl.model.HttpResponse.<init>(HttpMessage.scala:457)
[error] akka.http.scaladsl.model.HttpResponse$.apply(HttpMessage.scala:529)
[error] io.setu.caribou.grpc.deployment.DeploymentServiceHandler$.<clinit>(DeploymentServiceHandler.scala:24)
[error] io.setu.caribou.grpc.deployment.AbstractDeploymentServiceRouter.<init>(AbstractRouter.scala:21)
[error] routers.DeploymentRouter.<init>(DeploymentRouter.scala:29)

Akka gRPC creates handler classes with the following content. Line 24 is the line that defines the notFound response.

/*
 * Generated by Akka gRPC. DO NOT EDIT.
 */
object DeploymentServiceHandler {
    private val notFound = scala.concurrent.Future.successful(HttpResponse(StatusCodes.NotFound))
    private val unsupportedMediaType = scala.concurrent.Future.successful(HttpResponse(StatusCodes.UnsupportedMediaType))

If I run the test with testOnly routers.DeploymentRouterSpec, the test runs without issues. I've different routers each covered with a test. The issues is always the same. If i run all tests, all the akka gRPC related tests fails. If I run each gRPC related test independently, every test completes without an issue.

The root cause is, that the data: ByteString param defined on HttpEntity.Strict is null if I run all the tests.

[raboof]

Thanks for sharing, this is very interesting/puzzling.

It reminds me a lot of akka/akka-http#3221, which I haven't been able to reproduce so far.

It would be great if you could somehow help reliably recreate the issue.

[raboof]

The root cause is, that the data: ByteString param defined on HttpEntity.Strict is null if I run all the tests.

Right - but the question is how could it be null? I'm not sure what version of akka-http you're running, but in all versions the HttpResponse.apply that takes a status code should fill in HttpEntity.Empty as the entity, which is a Strict that has the (non-null) ByteString.empty as its data

[raboof]

Until we have a reproducer, let's at least collect other information: what versions of sbt, akka-http and Scala are you running?

[akkie]

The project the issue occurs is a commercial project. Currently I try to build a reproducer at home, but now I hit this issue: #946

The project at work uses Scala 2.13.2 and SBT 1.3.12. The Projects depends on Play 2.8.1 so it uses Akka HTTP 10.1.11.

[raboof]

Currently I try to build a reproducer at home, but now I hit this issue: #946

Perhaps the quick way out would be to run without HTTPS?

[akkie]

@raboof I've a reproducible test case: https://github.com/akkie/akka-grpc-1014

The test contains two test cases. The first ASpec calls CompactByteString.apply. The second FooSpec executes a gRPC test. The order of the tests is important. So if ASpec is executed before FooSpec then the issue occurs.

[raboof]

Great, that indeed reproduces it! I'll have a look.

[jrudolph]

👻 👻 👻 the mystery... the suspense...

[jrudolph]

The test contains two test cases. The first ASpec calls CompactByteString.apply. The second FooSpec executes a gRPC test. The order of the tests is important. So if ASpec is executed before FooSpec then the issue occurs.

Indeed, in ASpec, println(ByteString.empty) prints null.

[raboof]

Even a project with just akka-actor as a dependency:

object Main extends App {
  val byteString = CompactByteString.apply("test".getBytes)
  ByteString.empty.length
}

reproduces the problem. Definitely looks like an initialization order issue.

[jrudolph]

It is, I annotated running the object initializers, and dump the stack from where ByteString$.<clinit> is run:

CompactByteString
ByteString
java.lang.Exception: Stack trace
    at java.lang.Thread.dumpStack(Thread.java:1336)
    at akka.util.ByteString$.<clinit>(MyBS.scala:21)
    at akka.util.ByteString.<init>(MyBS.scala:754)
	at akka.util.CompactByteString.<init>(MyBS.scala:1041)
	at akka.util.ByteString$ByteString1C.<init>(MyBS.scala:179)
	at akka.util.ByteString$ByteString1C$.apply(MyBS.scala:164)
	at akka.util.CompactByteString$.<clinit>(MyBS.scala:1031)
	at test.ASpec.<init>(ASpec.scala:8)
    [snip]
Before ByteString.empty
After ByteString.empty
ByteString finished
CompactByteString finished

That line

    at akka.util.ByteString.<init>(MyBS.scala:754)

is

sealed abstract class ByteString
  extends IndexedSeq[Byte]
    with IndexedSeqOps[Byte, IndexedSeq, ByteString]
    with StrictOptimizedSeqOps[Byte, IndexedSeq, ByteString] {

  override protected def fromSpecific(coll: IterableOnce[Byte]): ByteString = ByteString(coll)
  override protected def newSpecificBuilder: mutable.Builder[Byte, ByteString] = ByteString.newBuilder
  override val empty: ByteString = ByteString.empty // <- this line introduces the cycle

[raboof]

@akkie so the short-term workaround could be to make sure you touch ByteString.empty before CompactByteString.apply, though we will of course work on fixing this on the Akka side :)

[jrudolph]

This line was introduced for the 2.13 version of ByteString so it makes sense that only 2.13 is affected.

The question is: are there other cycles that we don't know about?

[akkie]

@akkie so the short-term workaround could be to make sure you touch ByteString.empty before CompactByteString.apply, though we will of course work on fixing this on the Akka side :)

Thanks, the workaround works 👍

[raboof]

This is now also fixed upstream, so if you're on Akka 2.6.7 or later you should no longer need the workaround.