Parse +_ and -_ in types as identifiers under -Xsource:3 to support Scala 3.2 placeholder syntax

[neko-kai]

This change allows kind-projector plugin to rewrite +_ and -_ tokens to type lambdas,
in line with proposed syntax for Scala 3.2 in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html

See also https://scalacenter.github.io/scala-3-migration-guide/docs/tutorials/kind-projector.html (Soon on https://docs.scala-lang.org/scala3/guides/migration/compatibility-intro.html)

When used in conjunction with -Xsource:3 this will let the user use ? for wildcards and _ for placeholders, letting the user cross-compile the same sources with Scala 3 with -source:3.2 flag.

This change is not source breaking since currently +_ and -_ fail to parse entirely,
this change also does not allow the user to declare types with these names without backticks,
they can only be used as part of a type tree.

This does not add a special error when these symbols are used without kind-projector plugin – just "identifier not found", which is however in line with the behavior of other symbols currently used as placeholders in kind-projector – ? & *

See also:

• Original issue in kind-projector repository Scala 3 mode with _ as a placeholder and ? as wildcard? typelevel/kind-projector#120

[smarter]

This change allows kind-projector plugin to rewrite +_ and -_ tokens to type lambdas,
in line with proposed syntax for Scala 3.2 in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html

This page mentions _ but not +_ or -_, and I don't believe the latter two will become syntax in a future Scala 3 release, because we already disallowed variance annotations on type lambdas:

class Foo[F[+_]]
def foo(x: Foo[[+X] =>> Either[Int, X]]): Unit = {}
//              ^
//              no `+/-` variance annotation allowed here

This was disallowed because the compiler now infers the variance of a type lambda from its body, so the following works:

def foo(x: Foo[[X] =>> Either[Int, X]]): Unit = {}

Therefore in the future we can expect the following syntax to work too:

def foo(x: Foo[Either[Int, _]]): Unit = {}

But not a hypothetical syntax with +_.

[neko-kai]

@smarter

This page mentions _ but not +_ or -_, and I don't believe the latter two will become syntax in a future Scala 3 release, because we already disallowed variance annotations on type lambdas:

I was not aware of this change! However, they can probably be allowed to parse and be treated as just _ under -Ykind-projector for compatibility – along with +* and -*, that were omitted in initial -Ykind-projector PR – and that are required for cross-compiling variant type lambdas anyway, since 2.12 and 2.13 unfortunately do not infer variance by default.

The other glaring thing I was not aware of is that dotty doesn't actually implement underscore syntax right now. I was fooled by scastie! It was silently crashing on old versions of dotty when flag -source:3.2 was passed, giving the impression that source compiled correctly. There was never actually a -source:3.2 mode at all, let alone with underscores. 🤦‍♀️ (https://scastie.scala-lang.org/AAoaybtxQYaM40EsEl9gYA)

[smarter]

However, they can probably be allowed to parse and be treated as just _ under -Ykind-projector for compatibility – along with +* and -*, that were omitted in initial -Ykind-projector PR

That's one possibility, but it seems weird to me to introduce a new syntax that never existed in kind-projector under -Ykind-projector, the goal of which is to help existing code to compile, not to enable new ways to write code.

[neko-kai]

@smarter

That's one possibility, but it seems weird to me to introduce a new syntax that never existed in kind-projector under -Ykind-projector, the goal of which is to help existing code to compile, not to enable new ways to write code.

It seems weird now, but if this PR gets merged, then kind-projector will have such syntax – Scala 2 projects in the wild will use it and then this rule relaxation in -Ykind-projector will indeed help existing Scala 2 code to compile. Note that this would not enable new ways to write code, only accept existing Scala 2 constructs written in future syntax (that would have existed and made sense in Scala 3 if it was not obviated by variance inference).

[SethTisue]

It looks like this isn't actually gatewayed by -Xsource:3? Was that just an oversight?

You could argue that it doesn't matter, but especially since the new syntax is a no-op, it feels weird to me to simply accept it by default; I expected to have to opt-in.

[dwijnand]

Complicated matters, but I think I've understood the situation so this change LGTM.

@smarter am I right in taking your merging of the Dotty PR as agreeing to this change too?

[smarter]

The Dotty PR was about +* and -*, not about +_ and -_, I still stand by #9605 (comment), but I don't have strong feelings either way as long as I don't have to document this :).

[dwijnand]

Good point. With * deprecated in 3.2 and removed in 3.3, this PR looks to finding a way to compile code with Scala 2.13 with -Xsource:3 and Scala 3 with -source:3.2.

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

[smarter]

With * deprecated in 3.2 and removed in 3.3

Note that the timeline in http://dotty.epfl.ch/docs/reference/changed-features/wildcards.html is just there to give a rough idea of how things could go, it's not a fixed plan (in particular we may bump our minor version at any time just to signal that we broke forward binary compatibility).

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Yep.

[neko-kai]

@SethTisue

It looks like this isn't actually gatewayed by -Xsource:3? Was that just an oversight?

You could argue that it doesn't matter, but especially since the new syntax is a no-op, it feels weird to me to simply accept it by default; I expected to have to opt-in.

My reasoning for not gating was that it's not directly related to -Xsource:3 - instead it enables a kind-projector feature which may be used independently of -Xsource:3 (although without a ? substitute you would just lose any way to write a wildcard).
Although, I have no strong opinion and would gate or not depending on what you decide basically.

@dwijnand

Good point. With * deprecated in 3.2 and removed in 3.3, this PR looks to finding a way to compile code with Scala 2.13 with -Xsource:3 and Scala 3 with -source:3.2.

Yes, the timeline assumes a period when * is disabled, rendering cross-compilation impossible. My goal is to just jump straight to the end of the timeline and use _ everywhere in my libraries without losing cross-compilation (using appropriate future-enabling flags in builds).

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Well, I assumed Scala 3 already implements underscores, but I was wrong.
However there is a public plan to do this, so it will happen eventually, that's why I consider this change meaningful for cross-compilation even if the current RC or 3.0 final may not have underscores – because one of the next versions will eventually have them.

@smarter I can submit a quick PR that will allow use of _/+_/-_ as type lambda placeholders in -source:future using current the -Ykind-projector codepath. However this will not close scala/scala3#5379, because there's a question of what should F[_[_]] mean – in kind-projector this means [A[_]] =>> F[A], but in value-level lambdas f(_(_)) means f((a, b) => a(b)), so to avoid deciding anything on this question I would only be able to support *-kinded variables, in line with what -Ykind-projector supports right now.

By the way I've added the change to kind-projector typelevel/kind-projector#188 & verified that it works for my usecase 7mind/izumi#1479

[smarter]

I can submit a quick PR that will allow use of _/+_/-_ as type lambda placeholders in -source:future using current the -Ykind-projector codepath.

kind-projector repurposing _ right now would break a lot of existing code, to do that I think we'd need a deprecation period (in kind-projector or scala 2 itself under the -Xsource:3 flag) encouraging folks to switch to ? for wildcards.

[neko-kai]

kind-projector repurposing _ right now would break a lot of existing code, to do that I think we'd need a deprecation period (in kind-projector or scala 2 itself under the -Xsource:3 flag) encouraging folks to switch to ? for wildcards.

I thought Dotty's -source:future already disables _ for wildcards, so there is no syntax overlap?

For Scala 2 with typelevel/kind-projector#188 it repurposes _ only if the option -P:kind-projector:underscore-placeholders is passed, so my thinking is to get new syntax you would use:

• in Scala 2 - -Xsource:3 -P:kind-projector:underscore-placeholders
• in Scala 3 - -source:future

And if you pass no flags then nothing changes (And you'll get new syntax only in Scala 3.2/3.3 when it becomes the default)

[smarter]

I thought Dotty's -source:future already disables _ for wildcards, so there is no syntax overlap?

It only deprecates it for now.

it repurposes _ only if the option -P:kind-projector:underscore-placeholders

That could work for now yeah.

[neko-kai]

@smarter

It only deprecates it for now.

Got it, what about a new -Y flag, such as -Yunderscore-placeholders or a sub-option in -Ykind-projector such as -Ykind-projector:underscores?

I think that -source:3.2 could also work for that case, but there seems to have been a movement back to future instead of version numbers, and I don't think -source:future-future would work.

[smarter]

Honestly not sure, t'm wary of multiplying the flags since it makes everything more complicated, I guess we could go for the sub-option for now.

[neko-kai]

@smarter Submitted at scala/scala3#12378

[neko-kai]

I've decided to gate the change behind -Xsource:3 to minimize any possible disruption to the majority of existing code

[lrytz]

as I don't have to document this :)

Resonates with me :-) @neko-kai could you take this on? Would be really helpful to have a page that explains the status quo, the goal, and all the possible paths through the forest of combinations of

• (wildcard, placeholder) x
• (variance) x
• (2.13 2.13 -Xsource:3, 3.0, 3, 3 -source:3.1, 3 -source:3.2) x
• (-P:kind-projector:underscore-placeholders) x
• (-Yunderscore-placeholders)

@smarter where should that documentation go?

[smarter]

@smarter where should that documentation go?

Great question, maybe a dedicated page in the migration guide ? /cc @adpi2

[neko-kai]

@lrytz I've documented the Dotty part in the "Changes to wildcards", I've also expanded kind-projector README in the related kind-projector PR.
For scalac obviously this will be mentioned in release notes, but I don't know where to put the rest though (maybe just in comments or descriptions related to -Xsource:3 ?)

[lrytz]

Since this is mainly about migration to Scala 3, I think the migration guide is a good place. We can always link to it from elsewhere. It would be really good to have it all in one place.

I've documented the Dotty part in the "Changes to wildcards",

Where's that?

expanded kind-projector README

https://github.com/typelevel/kind-projector/pull/188/files?

[neko-kai]

@lrytz

Where's that?

https://github.com/lampepfl/dotty/pull/12378/files#diff-08f1a956e9a74ed84ae35bea06852879c16776956f075ec5f5b5748e117c6ecaR46
Here, but admittedly this probably deserves a separate page in migration guide

https://github.com/typelevel/kind-projector/pull/188/files?

Yep

[dwijnand]

So basically it only makes sense if both this PR and a Scala 3 "be allowed to parse and be treated as just _ under -Ykind-projector for compatibility" PR is merged, right?

Yep.

Got a verbal ok for that Scala 3 part being acceptable, so that's a re-✅ from me.

[neko-kai]

Migration guide updated at scalacenter/scala-3-migration-guide#203