Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Java 17 sealed in Java sources #10348

Merged
merged 1 commit into from May 1, 2023

Conversation

som-snytt
Copy link
Contributor

@som-snytt som-snytt commented Mar 15, 2023

Support sealed in Java sources. The related PR supports separate compilation, where PermittedSubclasses attribute specifies sealedness; there seems to be no ticket for that, alas. This PR is Java interop gravy.

Fixes scala/bug#12171
Fixes scala/bug#12159

@scala-jenkins scala-jenkins added this to the 2.13.12 milestone Mar 15, 2023
@SethTisue SethTisue modified the milestones: 2.13.12, 2.13.11 Mar 29, 2023
@SethTisue SethTisue added the prio:hi high priority (used only by core team, only near release time) label Mar 29, 2023
@SethTisue SethTisue requested a review from lrytz March 29, 2023 07:39
@SethTisue
Copy link
Member

@lrytz tentatively milestoned for 2.13.11 since it address a concern you raised on #10105

@som-snytt som-snytt marked this pull request as ready for review March 29, 2023 07:40
@lrytz lrytz force-pushed the followup/12159-mixed-java-sealed branch from f2a167c to 8751aeb Compare March 29, 2023 07:54
@som-snytt som-snytt force-pushed the followup/12159-mixed-java-sealed branch from 8751aeb to eeac512 Compare April 1, 2023 09:20
@som-snytt som-snytt changed the title Improve exhaustiveness check for mixed java sealed Improve exhaustiveness check for mixed java sealed [ci: last-only] Apr 1, 2023
@som-snytt
Copy link
Contributor Author

som-snytt commented Apr 1, 2023

The updated commit just "registers" Java classes and matches for completion after typer, where (per lrytz) either there is a permits or we look at classes from the same source file.

The simple impl could be improved or complexified further. Or simplified.

@som-snytt som-snytt marked this pull request as draft April 1, 2023 10:01
@som-snytt som-snytt force-pushed the followup/12159-mixed-java-sealed branch from eeac512 to 35c0384 Compare April 2, 2023 04:53
@lrytz
Copy link
Member

lrytz commented Apr 3, 2023

Thanks, great to see that we can make it work!

Why do you think we should delay until after typer? Forcing is what the typer does, I don't see a particular risk of running into cycles here, we don't need to infer anything to complete these class symbols.

For finding symbols defined in a particular compilation unit, I also think we need new global state, there doesn't seem to be anything we can reuse. A question is where to put it, I guess it doesn't matter too much. Here's a draft, what do you think (last commit)? 2.13.x...lrytz:scala:pr10348

@som-snytt
Copy link
Contributor Author

som-snytt commented Apr 3, 2023

Yes, that is what I was thinking of (a perRunCaches Map). I thought keeping the logic in one place was simpler, "children we know about" and "other offspring". Oh I see what you mean, we've already seen all the classes for this unit, nothing is gained by waiting.

The other piece I started looking at was emitting the permittedSubclasses attribute for Scala sealed. I'll try to add a commit for that, I was about to imitate some structure for innerClasses.

A final clean-up is also to review the tests for duplication or completeness.

Also, this is a patmat feature, so I like that it looks that way by putting the unwieldy sources map there.

@som-snytt
Copy link
Contributor Author

som-snytt commented Apr 3, 2023

I added your commits and will push after cleaning up my detritus.

Also I haven't touched the java parser to look more like your earlier edit, which I remember preferring.

@som-snytt som-snytt force-pushed the followup/12159-mixed-java-sealed branch from 35c0384 to a646610 Compare April 3, 2023 10:52
@@ -1266,6 +1273,11 @@ trait Namers extends MethodSynthesis {
val res = GenPolyType(tparams0, resultType)

val pluginsTp = pluginsTypeSig(res, typer, cdef, WildcardType)
cdef.getAndRemoveAttachment[PermittedSubclasses].foreach { permitted =>
clazz.updateAttachment[PermittedSubclassSymbols] {
PermittedSubclassSymbols(permitted.permits.map(typer.typed(_, Mode.NOmode).symbol))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to errors on bad permits clause here?

@som-snytt som-snytt marked this pull request as ready for review April 3, 2023 11:57
Copy link
Member

@lrytz lrytz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We forgot about inner classes...

package p;
public sealed interface X {
  public default int x() { return 1; }
}
final class Y implements X { }
final class O {
  final static class Z implements X { }
}
package p
class T {
  def n(a: X) = a match { case _: Y => 0 }
}

Should warn but doesn't in mixed compilation. Namer eagerly greates symbols for top-level classes. But symbols for inner classes are only created when the outer class completes.

So if there's a sealed Java type without a permits clause, we probably have to recursively force all classes defined in the same compilation unit.

X.java:9: error: anonymous classes must not extend sealed classes
    X bar = new X() { };
                    ^

😅 OK, at least that


@retronym brought runtime reflection to the table:

scala> import scala.reflect.runtime.{universe => ru}
import scala.reflect.runtime.{universe=>ru}

scala> ru.typeTag[p.X].tpe.typeSymbol.asClass.knownDirectSubclasses
val res0: Set[reflect.runtime.universe.Symbol] = Set()

Also macros / compiler plugins might want to see children, not only the exhaustivity checker. But I think it's fine to defer that.

@som-snytt som-snytt changed the title Improve exhaustiveness check for mixed java sealed [ci: last-only] Support Java 17 sealed in Java sources [ci: last-only] Apr 24, 2023
@som-snytt som-snytt force-pushed the followup/12159-mixed-java-sealed branch 2 times, most recently from 2bf8131 to 62c5acf Compare April 25, 2023 00:48
@som-snytt som-snytt changed the title Support Java 17 sealed in Java sources [ci: last-only] Support Java 17 sealed in Java sources Apr 25, 2023
@som-snytt
Copy link
Contributor Author

@lrytz this will be green after squashing. I guess "zucchini", as a green squash.

I'll take another look at the tweak to populating the sealed class hierarchy. It needs tests with non-sealed descendants.

Required for checking permitted subclasses
and for exhaustiveness of patterns.

Co-authored-by: Lukas Rytz <lukas.rytz@gmail.com>
@lrytz lrytz force-pushed the followup/12159-mixed-java-sealed branch from 9c3e87d to a0f546d Compare May 1, 2023 09:20
Copy link
Member

@lrytz lrytz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥒 ed

LGTM, thank you!

@lrytz lrytz merged commit bd9b10a into scala:2.13.x May 1, 2023
3 checks passed
@som-snytt som-snytt deleted the followup/12159-mixed-java-sealed branch May 1, 2023 14:32
@SethTisue SethTisue added the release-notes worth highlighting in next release notes label May 1, 2023
dongjoon-hyun pushed a commit to apache/spark that referenced this pull request Jun 19, 2023
### What changes were proposed in this pull request?
This PR aims to upgrade Scala to 2.13.11
- https://www.scala-lang.org/news/2.13.11

Additionally, this pr adds a new suppression rule for warning message: `Implicit definition should have explicit type`, this is a new compile check introduced by scala/scala#10083, we must fix it when we upgrading to use Scala 3,

### Why are the changes needed?
This release improves collections, adds support for JDK 20 and 21, adds support for JDK 17 `sealed`:
- scala/scala#10363
- scala/scala#10184
- scala/scala#10397
- scala/scala#10348
- scala/scala#10105

There are 2 known issues in this version:

- scala/bug#12800
- scala/bug#12799

For the first one, there is no compilation warning messages related to `match may not be exhaustive` in Spark compile log, and for the second one, there is no case of `method.isAnnotationPresent(Deprecated.class)` in Spark code, there is just

https://github.com/apache/spark/blob/8c84d2c9349d7b607db949c2e114df781f23e438/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L130

in Spark Code, and I checked `javax.annotation.Nonnull` no this issue.

So I think These two issues will not affect Spark itself, but this doesn't mean it won't affect the code written by end users themselves

The full release notes as follows:

- https://github.com/scala/scala/releases/tag/v2.13.11

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
- Existing Test
- Checked Java 8/17 + Scala 2.13.11 using GA, all test passed

Java 8 + Scala 2.13.11: https://github.com/LuciferYang/spark/runs/14337279564
Java 17 + Scala 2.13.11: https://github.com/LuciferYang/spark/runs/14343012195

Closes #41626 from LuciferYang/SPARK-40497.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
czxm pushed a commit to czxm/spark that referenced this pull request Jun 19, 2023
### What changes were proposed in this pull request?
This PR aims to upgrade Scala to 2.13.11
- https://www.scala-lang.org/news/2.13.11

Additionally, this pr adds a new suppression rule for warning message: `Implicit definition should have explicit type`, this is a new compile check introduced by scala/scala#10083, we must fix it when we upgrading to use Scala 3,

### Why are the changes needed?
This release improves collections, adds support for JDK 20 and 21, adds support for JDK 17 `sealed`:
- scala/scala#10363
- scala/scala#10184
- scala/scala#10397
- scala/scala#10348
- scala/scala#10105

There are 2 known issues in this version:

- scala/bug#12800
- scala/bug#12799

For the first one, there is no compilation warning messages related to `match may not be exhaustive` in Spark compile log, and for the second one, there is no case of `method.isAnnotationPresent(Deprecated.class)` in Spark code, there is just

https://github.com/apache/spark/blob/8c84d2c9349d7b607db949c2e114df781f23e438/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L130

in Spark Code, and I checked `javax.annotation.Nonnull` no this issue.

So I think These two issues will not affect Spark itself, but this doesn't mean it won't affect the code written by end users themselves

The full release notes as follows:

- https://github.com/scala/scala/releases/tag/v2.13.11

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
- Existing Test
- Checked Java 8/17 + Scala 2.13.11 using GA, all test passed

Java 8 + Scala 2.13.11: https://github.com/LuciferYang/spark/runs/14337279564
Java 17 + Scala 2.13.11: https://github.com/LuciferYang/spark/runs/14343012195

Closes apache#41626 from LuciferYang/SPARK-40497.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun pushed a commit to apache/spark that referenced this pull request Sep 16, 2023
### What changes were proposed in this pull request?
This PR aims to re-upgrade Scala to 2.13.11, after SPARK-45144 was merged, the build issues mentioned in #41943 should no longer exist.

- https://www.scala-lang.org/news/2.13.11

Additionally, this pr adds a new suppression rule for warning message: `Implicit definition should have explicit type`, this is a new compile check introduced by scala/scala#10083, we must fix it when we upgrading to use Scala 3

### Why are the changes needed?
This release improves collections, adds support for JDK 20 and 21, adds support for JDK 17 `sealed`:
- scala/scala#10363
- scala/scala#10184
- scala/scala#10397
- scala/scala#10348
- scala/scala#10105

There are 2 known issues in this version:

- scala/bug#12800
- scala/bug#12799

For the first one, there is no compilation warning messages related to `match may not be exhaustive` in Spark compile log, and for the second one, there is no case of `method.isAnnotationPresent(Deprecated.class)` in Spark code, there is just

https://github.com/apache/spark/blob/8c84d2c9349d7b607db949c2e114df781f23e438/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L130

in Spark Code, and I checked `javax.annotation.Nonnull` no this issue.

So I think These two issues will not affect Spark itself, but this doesn't mean it won't affect the code written by end users themselves

The full release notes as follows:

- https://github.com/scala/scala/releases/tag/v2.13.11

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
- Existing Test

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #42918 from LuciferYang/SPARK-40497-2.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prio:hi high priority (used only by core team, only near release time) release-notes worth highlighting in next release notes
Projects
None yet
4 participants