Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scalafix-cli_3 & test dependencies (GSoC 2022) #1650

Merged
merged 11 commits into from Oct 27, 2022

Conversation

rvacaru
Copy link
Contributor

@rvacaru rvacaru commented Jul 28, 2022

Follows #1629 & #1643
Towards #1680

Copy link
Collaborator

@bjaglin bjaglin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to see the first little bits of dotty in here!

However, compilation alone won't give many guarantees on this kind of code/PR, so I am wondering if we should

  1. tackle part of Break up unit the test monolith #1593 first to extract some tests relevant to -reflect to -reflect!
  2. just accept that we'll merge something that might be completely broken (and we'll figure it out only in one or 2 weeks when cross-compiling/running unit3)
  3. accept that this will be a larger PR than the others, including scalafix-reflect_3, scalafix-cli_3 and finally unit3 where we will know if this code works.

I would tend to favor (3), as long as this PR maintains a good git hygiene. Any other opinion?

Comment on lines 52 to 54
val defaultMsg = "Multiple compilation errors occurred for Scala 3, " +
"please try to use a CLI version which compiles into Scala 2"
Copy link
Collaborator

@bjaglin bjaglin Jul 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be quite abstract/hard to understand for an end-user.

What about something like:

Error compiling rule(s) from source using Scala 3 compiler; to use the Scala 2.x compiler instead, use the corresponding scalafix-cli artifact or force scalafixScalaBinaryVersion to 2.x in your build tool.

It's not perfect as scalafixScalaBinaryVersion is a sbt-scalafix concept, but

  1. judging by the number of tickets/requests, sbt-scalafix is by far the most used client so far
  2. build tools will have something similar (although maybe not named exactly the same) if they allow picking up a specific scalafix scala version
  3. we already have a reference to it in this repo
    s"To fix this problem, either remove ExplicitResultTypes from .scalafix.conf or make sure the scalafixScalaBinaryVersion setting key matches $inputBinaryScalaVersion."
    (although I don't remember having a proper discussion about it)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion on this tbh, I don't see any problem with requiring users to set a scalafixScalaBinaryVersion as well. Could be useful to document this config somewhere tho.

build.sbt Outdated Show resolved Hide resolved
@rvacaru
Copy link
Contributor Author

rvacaru commented Aug 4, 2022

Great to see the first little bits of dotty in here!

However, compilation alone won't give many guarantees on this kind of code/PR, so I am wondering if we should

  1. tackle part of Break up unit the test monolith #1593 first to extract some tests relevant to -reflect to -reflect!
  2. just accept that we'll merge something that might be completely broken (and we'll figure it out only in one or 2 weeks when cross-compiling/running unit3)
  3. accept that this will be a larger PR than the others, including scalafix-reflect_3, scalafix-cli_3 and finally unit3 where we will know if this code works.

I would tend to favor (3), as long as this PR maintains a good git hygiene. Any other opinion?

I don't like option 2 either, but then wouldn't it be the same for core3, rules3 that we've already merged? I just want to understand if there's a difference and what.
Between 1 and 3 my thought is what's gonna be more fun or where am I gonna learn more?

1 seems like creating a separate unit-reflect module and moving reflect tests from unit into this one. On Another note I'd like at some point to write some unit tests if possible (not the ones using input/output classes)

About 3 I don't mind the pr being bigger, but there's also ways to mitigate it via either branching off from this branch (reflect3) or by squashing commits and reviewing them one by one (eg. one commit for reflect3, one for cli3, etc)

@rvacaru rvacaru force-pushed the reflect3 branch 2 times, most recently from d18043e to 9a7c571 Compare August 5, 2022 12:30
@rvacaru rvacaru changed the title WIP - Scalafix-Reflect cross compiled to scala3 WIP - Scalafix-Reflect + Scalafix-cli cross compiled to scala3 Aug 5, 2022
build.sbt Show resolved Hide resolved
@rvacaru rvacaru changed the title WIP - Scalafix-Reflect + Scalafix-cli cross compiled to scala3 WIP - Scalafix-Reflect + Scalafix-cli + testkit cross compiled to scala3 Aug 5, 2022
build.sbt Outdated Show resolved Hide resolved
@rvacaru rvacaru force-pushed the reflect3 branch 3 times, most recently from 2f4e960 to 8dff964 Compare August 11, 2022 11:50
build.sbt Outdated Show resolved Hide resolved
@rvacaru rvacaru force-pushed the reflect3 branch 2 times, most recently from 7c30911 to c3456aa Compare August 12, 2022 15:21
@bjaglin
Copy link
Collaborator

bjaglin commented Oct 15, 2022

nearing merging state: down to 4 open, actionable comments

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 16, 2022

Rebased against 0.10.4

Just realized we had a major regression in ci-3 since that rebase.

It looks like ad2f544 triggers a (false positive?) failure in ScalafixArgumentsSuite that breaks testing infrastructure for further tests.

$ sbt "unit3Target3/testOnly *ScalafixArgumentsSuite"
...
[info] ScalafixArgumentsSuite:
[info] - ScalafixArguments.evaluate with a semantic rule !!! CANCELED !!! (3 milliseconds)
[info]   org.scalatest.exceptions.TestCanceledException was thrown. (ScalafixArgumentsSuite.scala:83)
[info] - ScalafixArguments.evaluate getting StaleSemanticdb !!! CANCELED !!! (0 milliseconds)
[info]   org.scalatest.exceptions.TestCanceledException was thrown. (ScalafixArgumentsSuite.scala:165)
[info] - ScalafixArguments.evaluate doesn't take into account withMode and withMainCallback !!! CANCELED !!! (0 milliseconds)
[info]   org.scalatest.exceptions.TestCanceledException was thrown. (ScalafixArgumentsSuite.scala:200)
Warning: Unable to read from client, please check on client for futher details of the problem.
Reporter completed abruptly with an exception after receiving event: TestSucceeded(Ordinal(0, 71),ScalafixArgumentsSuite,scalafix.tests.interfaces.ScalafixArgumentsSuite,Some(scalafix.tests.interfaces.ScalafixArgumentsSuite),CommentFileNonAtomic retrieves 2 patches,CommentFileNonAtomic retrieves 2 patches,Vector(),Some(97),Some(IndentedText(- CommentFileNonAtomic retrieves 2 patches,CommentFileNonAtomic retrieves 2 patches,0)),Some(LineInFile(257,ScalafixArgumentsSuite.scala,Some(Please set the environment variable SCALACTIC_FILL_FILE_PATHNAMES to yes at compile time to enable this feature.))),Some(scalafix.tests.interfaces.ScalafixArgumentsSuite),None,pool-1-thread-1-ScalaTest-running-ScalafixArgumentsSuite,1665911264396).
java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1286)
	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1577)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
	at org.scalatest.tools.SocketReporter.liftedTree1$1(SocketReporter.scala:46)
	at org.scalatest.tools.SocketReporter.apply(SocketReporter.scala:55)
	at org.scalatest.DispatchReporter.org$scalatest$DispatchReporter$Propagator$$_$run$$anonfun$1(DispatchReporter.scala:249)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:249)
	at java.lang.Thread.run(Thread.java:748)

Removing all cancel() in the suite seems to help... Will investigate

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 16, 2022

scalameta/scalameta#2825 seems to be culprit (incompatible with scalatest_3 3.2.{13,14} for some unknown reason), by the following bisect

https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/scalameta_2.13/

4.5.13+71-aca5dd6e-SNAPSHOT/ 	Tue Sep 20 07:02:07 UTC 2022 	OK
4.5.13+73-4db513a9-SNAPSHOT/ 	Tue Sep 20 11:37:57 UTC 2022 	KO  

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 16, 2022

I tried to force runtime scala libs to 2.13.8 in unit (to counter the transitive effect of bumping scalameta which now depends on them), without any success

    dependencyOverrides += "org.scala-lang" % "scala-compiler" % "2.13.8",
    dependencyOverrides += "org.scala-lang" % "scala-reflect" % "2.13.8",
    dependencyOverrides += "org.scala-lang" % "scalap" % "2.13.8",

I can't extract a repro and I can't get a proper stack trace, but I am wondering if it could be due to breaking changes brought by scala/bug#12650 and

scalafix/build.sbt

Lines 242 to 245 in 2708904

scalametaTeskit
.exclude("com.lihaoyi", "sourcecode_2.13")
.exclude("org.scala-lang.modules", "scala-collection-compat_2.13")
.exclude("org.scalameta", "munit_2.13"),

Next step would be to try a scalameta built against 2.13.10 to confirm/deny that hypothesis.

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 16, 2022

Next step would be to try a scalameta built against 2.13.10 to confirm/deny that hypothesis.

Using a local build of https://github.com/scalameta/scalameta/commits/6d9b59da427df20f79fa0786fc680e6596da1586 (which uses 2.13.10 to build) does fix the problem, so it does look related to scala/bug#12650. That means that scalameta 4.6.1 should help, so we could just wait for that.

I am still puzzled to know which part of the scalameta code is involved when using scalatest's cancel()... Any clue @tgodzik ?

@tgodzik
Copy link
Contributor

tgodzik commented Oct 17, 2022

I am still puzzled to know which part of the scalameta code is involved when using scalatest's cancel()... Any clue @tgodzik ?

No idea, it's super weird 🤔

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 17, 2022

Adding -Xlog:exceptions=info, we can see the deserialization issue (most likely on TestCanceledException since that's the exception thrown on the client-side)

$ sbt -J-Xlog:exceptions=info "unit3Target3/testOnly *ScalafixArgumentsSuite"
...
[70.108s][info][exceptions] Exception <a 'java/io/InvalidClassException'{0x00000000bf525de0}: local class incompatible: stream classdesc serialVersionUID = 1590627031578033034, local class serialVersionUID = 8944294921032069635>
 thrown in interpreter method <{method} {0x00007f3cc84a3ce0} 'readObject' '()Ljava/lang/Object;' in 'java/io/ObjectInputStream'>
 at bci 3 for thread 0x00007f3c2e92b000 (Thread-24)
[70.108s][info][exceptions] Exception <a 'java/io/InvalidClassException'{0x00000000bf525de0}: local class incompatible: stream classdesc serialVersionUID = 1590627031578033034, local class serialVersionUID = 8944294921032069635>
 thrown in interpreter method <{method} {0x00007f3b56599c20} 'react' '()V' in 'org/scalatest/tools/Framework$Skeleton$1$React'>
 at bci 15 for thread 0x00007f3c2e92b000 (Thread-24)

My assumption is that the client and the server sides of scalatest (per fork := true) run with a scala library that has significantly different bytecode for some core classes that TestCanceledException rely on, impacting the dynamic java serialVersionUID generation.

I still don't know why/how (see below), but it seems the scala library on the server-side is stuck to 2.13.8, while the one on the client-side is driven by scalameta:

  • 2.13.8 in 4.5.13 OK
  • 2.13.9 in 4.6.0 KO
  • 2.13.10 in 4.6.0-SNAPSHOT OK

Is the eviction effectively different depending on the JVM process (same classpath, different classloader) ?


With scalameta 4.6.0

  • Fork in 'console' task sbt/sbt#1918
    $ sbt unit3Target3/Test/console                                               
    ...
    Welcome to Scala 3.1.3 (11.0.16, Java OpenJDK 64-Bit Server VM).
    Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                            
    scala> println(util.Properties.versionString)
    version 2.13.8
    
  • println(util.Properties.versionString) reports "2.13.9" in tests
    $ sbt unit3Target3/Test/externalDependencyClasspath
    ...
    [info] * Attributed(/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.9/scala-library-2.13.9.jar)
    ...
    [info] * Attributed(/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scalap/2.13.8/scalap-2.13.8.jar)
    [info] * Attributed(/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.8/scala-reflect-2.13.8.jar)
    [info] * Attributed(/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar)
    

@bjaglin
Copy link
Collaborator

bjaglin commented Oct 17, 2022

Is the eviction effectively different depending on the JVM process (same classpath, different classloader) ?

That's indeed the problem.

sbt uses a layered classloader by default, which loads scala libraries defined in scalaInstance ahead, independently of the user classpath(s).

$ sbt "show unit3Target3/scalaInstance"
[info] Scala instance { version label 3.1.3, actual version 3.1.3, library jars:
/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.1.3/scala3-library_3-3.1.3.jar,
/home/brice/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar,
...

This library is different than the one brought transitively by scalameta-testkit 4.6.0 in the user classpath (2.13.9), which does get loaded in the forked process. Because of the breaking change in 2.13.9, serialization is impacted and the 2 processes can't communicate through classes serialized through a dynamically-generated serialVersionUID 🤦

This problem only happens with unit3 (unit2_13 is fine) as overrideScalaVersion does not seem to be honored with Scala 3.

Options by order of preference (but maybe not speed to unblock this PR)

  1. wait for scalameta to be published with something else than scala-library:2.13.9
  2. Implement manually overrideScalaVersion: force the Scala 2 library version from scalaInstance (as brought by the Scala 3 library version) into the user classpath
  3. use ClassLoaderLayeringStrategy.Flat to have scala-library:2.13.9 on both scalatest client and server (perf regression expected)
  4. disable forking (other problems expected)

I'll add a commit with (3) for now to continue addressing other comments of this PR.

rvacaru and others added 6 commits October 18, 2022 12:33
Using scala.reflect.io in Scala 3 is a recipe for bugs as we should be
using the equivalent dotty classes instead
In scala 3's AbstractFileClassLoader, `loadClass` is routing loadClass
calls to findClass, causing "attempted duplicate class definition"
when loading a rule sharing the same FQN as a previously compiled one.

In scala 2, there is no failure but loaded rules could be the result
of previous compilations, so returning a fresh classloader each time
is probably what we want.
Comment on lines +63 to +66
"Error compiling rule(s) from source using Scala 3 compiler; " +
"to use the Scala 2.x compiler instead, use the corresponding " +
"scalafix-cli artifact or force scalafixScalaBinaryVersion " +
"to 2.x in your build tool"
Copy link
Collaborator

@bjaglin bjaglin Oct 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't manage to get that error when trying to compile a rule with quasiquotes with scalafix-cli_3

cs launch -r ivy2local ch.epfl.scala:scalafix-cli_3.1.3:0.10.4+22-b8cd292e+20221018-1241-SNAPSHOT -M scalafix.cli.Cli -- --rules=https://raw.githubusercontent.com/scalacenter/scalafix-named-literal-arguments/fb9f65ee9f09f167651b3d9bfe94a9be76818759/scalafix/rules/src/main/scala/fix/Namedliteralarguments_v1.scala
error: 2 errors
[E0]  Any is not a valid result type of an unapply method of an extractor.
[E1] Scala 2 macro cannot be used in Dotty. See https://dotty.epfl.ch/docs/reference/dropped-features/macros.html
To turn this error into a warning, pass -Xignore-scala2-macros to the compiler

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added d15af9a

cs launch -r ivy2local ch.epfl.scala:scalafix-cli_3.1.3:0.10.4+23-d15af9a0+20221018-1320-SNAPSHOT -M scalafix.cli.Cli -- --rules=https://raw.githubusercontent.com/scalacenter/scalafix-named-literal-arguments/fb9f65ee9f09f167651b3d9bfe94a9be76818759/scalafix/rules/src/main/scala/fix/Namedliteralarguments_v1.scala
error: 3 errors
[E0]  Any is not a valid result type of an unapply method of an extractor.
[E1] Scala 2 macro cannot be used in Dotty. See https://dotty.epfl.ch/docs/reference/dropped-features/macros.html
To turn this error into a warning, pass -Xignore-scala2-macros to the compiler
[E2] Error compiling rule(s) from source using Scala 3 compiler; to use the Scala 2.x compiler instead, use the corresponding scalafix-cli artifact or force scalafixScalaBinaryVersion to 2.x in your build tool

@bjaglin bjaglin marked this pull request as ready for review October 18, 2022 11:22
@bjaglin bjaglin merged commit b3d18fe into scalacenter:main Oct 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants