Possible change in dependency publication method or in dependency solving method

[TonioGela]

Version(s)
1.2.0

Describe the bug
Sorry for the possibly vague and confused issue title, but I'm confused as well.
In the typelevel toolkit CI we rely on the org.virtuslab.scala-cli::cli dependency to run unit tests against all the (versions, platform) combinations we support invoking scala cli directly and not relying on the command line launcher, a process that I also described here as it was really fun to implement.

Since the cli_2.13 artifact isn't available for most of the scala cli versions we've always relied on cli_3, using CrossVersion.for2_13Use3 (here)

Since 1.2.0, we're getting a dependency resolution error:

 [error]  Error downloading org.virtuslab.scala-cli:test-runner_2.13:1.2.0
  not found: /home/runner/.ivy2/local/org.virtuslab.scala-cli/test-runner_2.13/1.2.0/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/org/virtuslab/scala-cli/test-runner_2.13/1.2.0/test-runner_2.13-1.2.0.pom
  No fallback URL found

Taking a look on maven central I've noticed that test-runner_2.13 isn't published for 1.2.0, but neither is runner_2.13, so I can't really tell where the dependency resolution problem is (or even if there is one). What confuses me the most is what I wrote in the To Reproduce section later.

To Reproduce
I've minimized the (possible) bug here:

(1.1.3, calling "compile") WORKS:

//> using scala 2.13.12
//> using option -Ytasty-reader
//> using dep org.virtuslab.scala-cli:cli_3:1.1.3

import scala.cli.ScalaCli
object Main {
    def main(args: Array[String]): Unit = ScalaCli.main(Array("compile", "."))
}

(1.1.3, calling "test") WORKS:

//> using scala 2.13.12
//> using option -Ytasty-reader
//> using dep org.virtuslab.scala-cli:cli_3:1.1.3

import scala.cli.ScalaCli
object Main {
    def main(args: Array[String]): Unit = ScalaCli.main(Array("test", "."))
}

(1.2.0, calling "compile") WORKS:

//> using scala 2.13.12
//> using option -Ytasty-reader
//> using dep org.virtuslab.scala-cli:cli_3:1.2.0

import scala.cli.ScalaCli
object Main {
    def main(args: Array[String]): Unit = ScalaCli.main(Array("compile", "."))
}

(1.2.0, calling "test") FAILS:

//> using scala 2.13.12
//> using option -Ytasty-reader
//> using dep org.virtuslab.scala-cli:cli_3:1.2.0

import scala.cli.ScalaCli
object Main {
    def main(args: Array[String]): Unit = ScalaCli.main(Array("test", "."))
}

and the reported error is

Downloading 2 dependencies and one internal dependency
[error]  Error downloading org.virtuslab.scala-cli:test-runner_2.13:1.2.0
  not found: /Users/toniogela/.ivy2/local/org.virtuslab.scala-cli/test-runner_2.13/1.2.0/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/org/virtuslab/scala-cli/test-runner_2.13/1.2.0/test-runner_2.13-1.2.0.pom
  No fallback URL found

I should also report that switching to the latest scala 3 version makes the four snippet compile and run successfully, so it seems related to the way deps are resolved for scala 2.13, but just for tests as "compile" works correctly despite runner_2.13 version 1.2.0 was NOT published.

Also, ofc, running scala-cli test X using the command line launcher where X is a .test.scala file containing the directive //> using scala 2.13.12 works flawlessly (and this adds even more confusion)

Expected behaviour
I won't claim that I expect all four aforementioned snippets to work flawlessly, as this is clearly a non-standard use case that relies on dependencies that weren't meant to be used by the general public, but I would love to be enlightened about what it might be the possible root cause, and if this is due to a design decision that we should cope with to keep testing the typelevel toolkit.

Thanks in advance!
//cc @armanbilge

[Gedochao]

Starting with 1.2.0 we cross compile most module sof Scala CLI with Scala 3 LTS & Next (so right now 3.3.3 & 3.4.0), while publishing the LTS artifacts.
I suspect this is an unintentional result of the large refactor tied to that, likely somewhere in the changes of #2752 or #2759.
I can see the test-runner_3 dependency was published, while its 2.13 equivalent wasn't: https://repo1.maven.org/maven2/org/virtuslab/scala-cli/test-runner_3/1.2.0/
You are also correct the runner artifact wasn't published for 1.2.0...

@TonioGela TL;DR it seems that this is unintentional and we didn't catch it since technically speaking, those are internal APIs, not necessarily prepared for external use (so there's no guarantees on our side, really).
However, I see no reason why not to publish those artifacts, we'd just need to tweak the implementation to publish them along with the next version.

[Gedochao]

It seems that runner wasn't published, since it's compiled with a different Scala versions (3.0.2) for compatibility reasons, which makes the publishing script skip it while publishing (since it only publishes LTS artifacts now).
This is the likely culprit.

@TonioGela what you need working is the cli module and all its dependencies, correct?
I can try to fix it before v1.2.1 comes out... (mentioning #2790 to not forget about it)
Do you also need the _2.13 artifacts?

[Gedochao]

So I've been wondering why would test-runner_2.13 be necessary when you're using cli_3 as the dependency... as I'd expect it to fetch test-runner_3 as well.

scala-cli/modules/options/src/main/scala/scala/build/Artifacts.scala

Line 114 in 178281e

Seq(dep"$testRunnerOrganization::$testRunnerModuleName:$testRunnerVersion")

Sigh.
You're using cli_3, but the test runner follows the actual Scala version used in the app.

[TonioGela]

@TonioGela what you need working is the cli module and all its dependencies, correct? I can try to fix it before v1.2.1 comes out... (mentioning #2790 to not forget about it) Do you also need the _2.13 artifacts?

@Gedochao sorry for the late response and first of all, thanks for your support! (also since my issue wasn't clear at all)

I'm not sure we strictly need _2.13 artifacts, unless (like you mentioned) cli_3 pulls it from maven central when running the tests of a 2.13 script.

In any case, any fix will be more than welcome, as we could continue testing successfully.

I saw that you've closed it here, is there any snapshot I can try to use to see whether your change fixes our problem too?

[Gedochao]

is there any snapshot I can try to use to see whether your change fixes our problem too?

@TonioGela sure, you can try 1.2.0-35-gc05e9bd1e-SNAPSHOT... or just wait a little bit, as Scala CLI v1.2.1 is being released right now.

I believe everything should work with v1.2.1. If you run into any problems, please reopen this issue.

unless (like you mentioned) cli_3 pulls it from maven central when running the tests of a 2.13 script

Yes, I believe this is the case.

[TonioGela]

Take a look here @Gedochao: https://github.com/typelevel/toolkit/actions/runs/8482335078/job/23241385459
It's working flawlessly! I'll wait till 1.2.1 to update the dependency ofc, but thanks infinitely for your help!