0.13.12 regression in resolving org.scala-lang:scala-library in custom configuration

[olafurpg]

steps

Repo to reproduce: https://github.com/olafurpg/scalafmt/files/521826/issue485.zip

problem

Running scalafmt results in errors

[error] Modules were resolved with conflicting cross-version suffixes in {file:/Users/heiko/projects/scala/new-in-scala-212/}new-in-scala-212:
[error]    org.scala-lang.modules:scala-xml _2.11, _2.12.0-RC1

OR

java.lang.NoClassDefFoundError: scala/Product$class
    at org.scalafmt.config.Docstrings$ScalaDoc$.<init>(Docstrings.scala:8)
    at org.scalafmt.config.Docstrings$ScalaDoc$.<clinit>(Docstrings.scala)
    at org.scalafmt.config.ScalafmtConfig$.apply$default$2(ScalafmtConfig.scala:2)
    at org.scalafmt.config.Settings$class.$init$(Settings.scala:17)
    at org.sc

I am unable to reproduce the issue with sbt.version=0.13.11 in build.properties.

I tried the following with no success:

dependencyOverrides += "org.scala-lang" % "scala-library" % org.scalafmt.Versions.scala % "scalafmt"
// and
libraryDependencies += "org.scala-lang" % "scala-library" % org.scalafmt.Versions.scala % "scalafmt" force()

However, changing to typelevel/scala fixes the issue (PR: https://github.com/olafurpg/scalafmt/pull/522):

libraryDependencies += "org.typelevel" % "scala-library"     % org.scalafmt.Versions.scala   % "scalafmt"

expectation

I expected the scalafmt task to run without error.

More specifically, I expect the 0.13.12 resolver to not evict org.scala-lang:scala-library:2.11.8 from the scalafmt configuration in favor of org.scala-lang:scala-libary:2.12.0-RC1 defined in the default/compile configuration.

notes

• Potential cause of this issue: Override scala organization and version transitively at the Ivy level #2634
• Original scalafmt issue: https://github.com/olafurpg/scalafmt/issues/485#issuecomment-253978435
• Scalafmt is 2.11 only. To overcome the 2.10 requirement for SBT plugins, the scalafmt SBT plugin uses a custom configuration to classload scalafmt with 2.11.8 scala-library.
• Attempts to port scala.meta (a scalafmt dependency) to 2.10 have so far been unsuccessful: port scala.meta to 2.10 scalameta/scalameta#295

[eed3si9n]

How about with 0.13.13-RC3?

[olafurpg]

Good question. Unfortunately, I get the same error:

[info] java.lang.NoClassDefFoundError: scala/Product$class
[info]  at org.scalafmt.config.Docstrings$ScalaDoc$.<init>(Docstrings.scala:8)
[info]  at org.scalafmt.config.Docstrings$ScalaDoc$.<clinit>(Docstrings.scala)

[eed3si9n]

@milessabin Could this be related to the mediator?

[milessabin]

Yes, it could, but so far I've not been persuaded that what's being attempted makes any sort of sense at all ... in what circumstances is it reasonable to want more than one version of scala-library on the classpath?

[olafurpg]

To my best understanding, the classpaths are isolated between different configurations so there will only be one version of scala-library. Here's what the SBT documentation says about configurations:

Ivy configurations are a useful feature for your build when you need custom groups of dependencies, such as for a plugin.

It may very well be that I'm using configurations for the wrong purpose. All I want is a place to put together dependencies so I can classload scalafmt with scala-library 2.11.8. I am open for any suggestions on how to accomplish that.

PS. The classpath in https://github.com/olafurpg/scalafmt/pull/522 (where I used org.typelevel:scala-library) contained two versions of scala-library. I agree that is a terrible idea. I closed the PR.

[olafurpg]

More users have reported this issue. My current recommendation to them is to downgrade to 0.13.11, which I think is a shame.

Is there a chance this might get fixed in 0.13.13? I would be happy with the ability to force a scala-library version in a specific scope:

libraryDependencies += "org.scala-lang" % "scala-library" % "2.11.8" % "scalafmt" force()

Alternatively, is there another way to call 2.11 libraries in SBT plugins?

[milessabin]

I'd like to help, but I have no idea what the implications of calling 2.11 libraries in SBT plugins are. All my experience with Scala over the last decade tells me that mixing Scala versions in a single JVM can't be expected to work.

[sjrd]

It can be expected to work if you do this in completely different class loaders. And class loaders created from classpaths in different configurations should be expected to work if those alternative configurations do not have a stray scala-library.jar. If sbt is mixing up scala-library.jar across configurations, this is a bug and it should be fixed. Configurations should be completely isolated from each other in terms of dependency resolution and classpath construction.

[sjrd]

In Scala.js, we use a similar mechanism to depend on a specific version of Jetty for use (via classloader) by our sbt plugin. See here: https://github.com/scala-js/scala-js/blob/v0.6.13/sbt-plugin/src/main/scala/org/scalajs/sbtplugin/ScalaJSPluginInternal.scala#L948-L962
This allows us to work even if another sbt plugin on the classpath depends on Jetty 9. This is essentially the same issue as using different Scala versions: you cannot mix Jetty 8 code and Jetty 9 code in the same classloader; but in different class loaders, it's fine.

The fact that our mechanism works for the jetty dependency, and that scalafmt's mechanism stopped working for the scala-library dependency, is a clear indicator that sbt is messing with scala-library in ways it shouldn't do.

[sjrd]

And to finish with a more constructive comment (instead of sounding like a rant), the above:

sbt is messing with scala-library in ways it shouldn't do.

means that this thing is definitely fixable ;) And that's the direction in which we should look to solve the issue.

[eed3si9n]

Ivy configurations are meant to be a dependency graph on its own, so if the plugin authors create a sandbox configuration that does not depend on Runtime, Compile or Test it's supposed to be isolated from everything else, and should be safe to use any (or even multiple) versions of Scala.

That does conflict with the general design goal of the mediator added by Miles, which enforces scalaVersion.value down to scala-* artifacts. We could:

• check the scala-* artifact is binary compatible with scalaVersion first, and only when it's binary compatible enforce the scalaVersion.
• or, enforce the scalaVersion only when the configuration is a descendant of Runtime, such as Compile, Test, and any custom configuration that extends them.

What do you think?

[dwijnand]

I think I prefer the first option, as I think when you have an error because you have different major Scala versions it'll be more obvious - as opposed to a configuration-based decision.

eg. I think it'll be easier to understand "here's the problem, I'm using a dependency the uses Scala 2.12", as opposed to "oh right, yeah, I should've remembered to extend Runtime"

[dwijnand]

The second option also special-cases the topology of configurations descendant from Runtime. Is it correct to assume that we want to only align scalaOrganization and scalaVersion across the dependency graph for just Runtime or should it be true for any custom topology?

A third option is what Olaf suggested: honour when force() is called.

[eed3si9n]

@dwijnand We already special case Compile and Test to do all sorts of things like adding scala-library in there, so I don't think it's particularly odd that the scalaVersion enforcement is limited towards them.

We can introduce a new setting called scalaVersionConfigurations or something like that and default it to Compile (or Runtime) descendants, and the build users can tweak it if she wants a sandbox config to have the scalaVersion enforced.

[dwijnand]

If we want to go the configuration way, I think that's a good idea (wiring that down to the mediator, with bincompat in mind, might be really tricky though).

Do you think the force() idea is bad?

[milessabin]

I think @eed3si9n's first option defeats a large part of the purpose of the mediator, so I find the second option a lot more appealing.

[milessabin]

I'm not quite getting the force() suggestion ... how is that restricted to a particular configuration?

I think that any mechanism which mixes different versions of scala- artefacts within a single process should be done via classloader isolation and distinct configurations (if it's done at all).

[eed3si9n]

I'm not feeling the force() because that means sandbox configuration is broken by default for sbt 0.13.12+, but works fine on 0.13.11.

[sjrd]

Could someone fill me in/point me to some info on this mediator thing.

[dwijnand]

@sjrd See #2634

https://github.com/sbt/sbt/pull/2634/files#diff-633274194cf279e704380f18f102f28aR60

[eed3si9n]

@sjrd It's a feature that was added in sbt 0.13.12 (#2634) that overrides scala organization and scala version transitively for scala-* artifacts like scala-compiler and scala-library.

[dwijnand]

@milessabin I don't think it does defeat a large part: the problem in #2286 was about using dependencies that transitively depended on for example scala-reflect or scala-compiler 2.11.5 (because that was the version of scala used when they were published) while having scalaVersion set to 2.11.8. The scope was always to upgrade the minor version (x.y.Z) of the scala artefacts.

But I don't see anyone against the configuration idea, so maybe that's the best choice.

[sjrd]

I see, thanks. IMO the configuration-based idea is better, in that case.

[jvican]

It's a shame this issue was not fixed before 0.13.13. What's the next version that will address this bug?

[eed3si9n]

I guess we sort of agreed on the direction of the fix but we never assigned who's going to work on it. Once we have a fix, I'd be happy to publish 0.13.14-M1.

[hseeberger]

Please fix asap, else scalafmt cannot be used for Scala 2.12.

[olafurpg]

Would it be possible to use sbt 0.13.13 synthetic projects as a replacement for this custom-scope + update hack? Instead of adding a library dependency in a custom scope inside all projects, the plugin could create a synthetic project myPluginProject with a regular dependency on a 2.11/2.12 library. The plugin could then run the myPluginProject/runMain xxx task to invoke the library as a replacement of running update in the custom scope and classload from that classpath.

Given that 2.10 is almost three years old now and two major versions away from the latest scala release, I think it would be nice if sbt offered a clean story for plugins to use modern scala versions.

BTW, I've decided to remove the scalafmt sbt plugin in its current form https://github.com/olafurpg/scalafmt/issues/597

[dwijnand]

It might be a way to workaround this regression.

Do users of the sbt-plugin tend to format everything in one go, or on a project-by-project basis?

[olafurpg]

Do users of the sbt-plugin tend to format everything in one go, or on a project-by-project basis?

I think the most common use-case is to format everything in one go. However, does it matter? To only format a subproject, the sbt plugin can pass in different flags to the mySyntheticPluginProject/runMain --flag subproject/.* task.

[olafurpg]

The next release of the scalafmt sbt plugin will use synthetic projects as a workaround for this issue, see https://github.com/olafurpg/scalafmt/pull/610. The plugin ended up being much smaller than the older plugin, which I'm very happy about.

sbt 0.13.13 is nice, thanks for a fantastic release. Feel free to close this issue.

[sjrd]

Feel free to close this issue.

I wouldn't close this issue simply because there exists a workaround that takes a completely different direction. The fact remains that this is a bug, and it could affect other people.

[dwijnand]

Yeah, it's still a bug. Not closing it.

[liufengyun]

I hit on a related issue when working with Dotty + ScalaTest: scala/scala3#5612

[eed3si9n]

@liufengyun I thought #2828 fixed this issue in 2016.
If Dotty is going to reverse the assumption made by #2634 that says organization and version number have to line up for scala-compiler, scala-library, scala-reflect, scala-actors, and scalap, we should open a separate issue to discuss this.