For Android compatibility, make Statics.releaseFence() also catch NoSuchMethodException for java.lang.invoke.VarHandle.releaseFence() call

[nwk37011]

Currently Statics.releaseFence() call catches ClassNotFoundException only for the java.lang.invoke.VarHandle.releaseFence() call. However, there are cases where java.lang.invoke.VarHandle exists but not releaseFence() such as Android, this commit tries to handle such cases more properly by catching NoSuchMethodException thus allowing such systems to fallback to sun.misc.Unsafe.storeFence()

[NthPortal]

LGTM. I don't see how this could break anything. still would definitely like to hear Jason's thoughts

[lrytz]

Looks safe to me too, but @nwk37011 do we know if this actually fixes releaseFence on Android? At https://users.scala-lang.org/t/scala-2-13-on-android/7235/24, I read

Android explicitly notifies the method is blacklisted so it’s bit uncertain if catching NoSuchMethodException to route to the fallback will work if Android is just shutting down VM for trying to poke the method.

[SethTisue]

review by @makingthematrix perhaps? (and/or, can you suggest anyone else with Android knowledge who might like to take a look?)

[makingthematrix]

review by @makingthematrix perhaps? (and/or, can you suggest anyone else with Android knowledge who might like to take a look?)

Looks good to me. If you want me to test it, I can maybe do it on Sunday.
And maybe @olafurpg could also take a look.
My test would be to build this app without Olafur's maven's plugin which I use for now to fix this bug.

[lrytz]

What is Olaf's plugin and how does it work around the issue?

[makingthematrix]

What is Olaf's plugin and how does it work around the issue?

I don't know exactly but here's the code and here's the discussion about it.

[nwk37011]

Looks safe to me too, but @nwk37011 do we know if this actually fixes releaseFence on Android? At https://users.scala-lang.org/t/scala-2-13-on-android/7235/24, I read

Android explicitly notifies the method is blacklisted so it’s bit uncertain if catching NoSuchMethodException to route to the fallback will work if Android is just shutting down VM for trying to poke the method.

The end of the discussion leads to a github repo with Statics.java patched in which does exactly the same thing.

https://github.com/androidscala213/AndroidScala213/blob/master/scala-library-hack/src/main/java/scala/runtime/Statics.java

From own testing of the code, Android gives a log of java.lang.invoke.VarHandle.releaseFence() being blacklisted along with NoSuchMethodException and across various Android SDK versions catching it enabled to fall back to calling sun.misc.Unsafe.storeFence() without any problem so I believe it must be pretty safe. The worst possible case I can imagine is possibly becoming just as the same as how it was before, returning ExceptionInInitializerError, if somehow google decides to explicitly block sun.misc.Unsafe.storeFence() in future Android releases. However, sun.misc.Unsafe.storeFence() has been around for pretty long time and according to https://developer.android.com/guide/app-compatibility/restrictions-non-sdk-interfaces it is stated as not blacklisted across many versions of releases so I doubt if it will happen anytime soon

[lrytz]

I'm happy to merge this, but it'd be great if @makingthematrix you could test it.

If it makes testing easier, we can merge this now so that an integration build will be available on https://scala-ci.typesafe.com/ui/native/scala-integration.

[makingthematrix]

If it makes testing easier, we can merge this now so that an integration build will be available on https://scala-ci.typesafe.com/ui/native/scala-integration.

That would be great. Thank you!

[lrytz]

Version 2.13.7-bin-768001e is available (https://stackoverflow.com/questions/40622878/how-do-i-tell-sbt-to-use-a-nightly-build-of-scala-2-12-or-2-13)

[makingthematrix]

@lrytz
Okay, I tested 2.13.7-bin-768001e. It works. But, what's weird, also the standard 2.13.6 works now. It could be that in the meantime changes in another piece of the puzzle (Scala 2.13.5 -> 2.13.6? GraalVM Native Image? Gluon Substrate? JavaFX?) made this change unnecessary - at least in my case.

On the other hand, it also doesn't make anything worse. Yay, I guess?...

[lrytz]

There are a lot of pieces in this puzzle I don't know of, or anything about :-) I guess you made sure to remove the workaround(s), like svm-subs?

Maybe no workarounds are needed anymore with recent GraalVM versions? oracle/graal#2761 (comment) says

Full method handles support will be included in the 21.0 release

The original description of this PR says

VarHandle exists but not releaseFence on Android

I actually don't understand yet what this means - @nwk37011 / @makingthematrix could you clarify?

• Is this referring to the JVM-like VM in Android, which doesn't support releaseFence? Then I don't understand why we're talking about GraalVM native image.
• Or are you both talking about running Scala on Android as native code through GraalVM? (Does Android support running native code?) Then the lack of support for releaseFence is native image and has nothing to do with Android, right?

[makingthematrix]

@lrytz :
Yes, I tested a few of example apps from my Scala on Android repo. I used the Scala build you linked to and removed the svm-subs plugin, but also tried with Scala 2.13.6 and without the plugin. As I understand, this problem may appear in a few different situations. In my case, I use GraalVM Native Image to build Android apps, and they crashed with an error related to releaseFence when I used GraalVM 20.x without svm-subs. So I guess it was fixed on the GraalVM side in 21.x and now I don't need svm-subs anymore.

But that means I can test this once again, this time with an older GraalVM. I will do it and come back to you.

The code snipped linked by @nwk37011 a few comments above is from a different attempt to build Android apps with Scala, in a more standard way, without GraalVM Native Image.

[nwk37011]

@lrytz
Yes, the most obvious case is of Android and is not really about GraalVM. Android does not return ClassNotFoundException for poking java.lang.invoke.VarHandle, which suggests the class may do exist. It does not however, allow calling VarHandle.releaseFence() and returns NoSuchMethodException instead. Scala 2.11.x did not utilize VarHandle.releaseFence() extensively as far as I know thus most app written in Scala used Scala 2.11.x with sbt-android plugin https://github.com/scala-android/sbt-android which was the last available Scala version to Android so far. Android build tools actually can do understand Scala 2.12.x and 2.13.x library and compile it along with the introduction of newer R8 compiler which is the default compiler at this point.

[lrytz]

OK, so it's actually possible to run Java 8 bytecode on android these days, officially supported by Google. I wasn't aware, that's awesome!

IIUC https://developer.android.com/studio/write/java8-support.html#supported_features

• d8 desugars invokedynamic to bytecote that the Android Runtime (ART) understands
• r8 (seems to be a fork of proguard?) takes the .class files of the app and all dependencies, does the tree-shaking and bundling into a single .dex file

I played around a little and managed to run d8 on the 2.13.6 standard library. Downlaoded https://developer.android.com/studio#cmdline-tools

$> java -cp /Users/luc/Downloads/cmdline-tools/lib/r8.jar com.android.tools.r8.D8 --output o /Users/luc/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar --classpath /Users/luc/.jabba/jdk/adopt@8/Contents/Home/jre/lib/rt.jar

there are a bunch of Malformed inner-class attribute warnings, and at the end, interestingly:

Error in /Users/luc/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar:scala/runtime/Statics.class at Lscala/runtime/Statics;releaseFence()V:
com.android.tools.r8.internal.m1: MethodHandle.invoke and MethodHandle.invokeExact are only supported starting with Android O (--min-api 26)
Compilation failed
Exception in thread "main" java.lang.RuntimeException: com.android.tools.r8.CompilationFailedException: Compilation failed to complete, position: Lscala/runtime/Statics;releaseFence()V, origin: /Users/luc/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar:scala/runtime/Statics.class

The same happens with the scala-library-2.13.7-bin-768001e.jar (after this PR). But anyway, adding --min-api 26 helps and makes the conversion pass, producing a classes.dex.

I converted this back to a .jar with https://github.com/pxb1988/dex2jar, and indeed the invokedynamic LambdaMetaFactory calls are desugared into anonymous classes.

---

Anyway, nothing else to be done here, we can keep the change of this PR. It's really great to have multiple options available for Scala on Android (native code through GraalVM, translation of classfiles like Kotlin / Java 8, Scala.js and React Native).