Skip to content

Commit

Permalink
[SPARK-40497][BUILD] Re-upgrade Scala to 2.13.11
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This PR aims to re-upgrade Scala to 2.13.11, after SPARK-45144 was merged, the build issues mentioned in #41943 should no longer exist.

- https://www.scala-lang.org/news/2.13.11

Additionally, this pr adds a new suppression rule for warning message: `Implicit definition should have explicit type`, this is a new compile check introduced by scala/scala#10083, we must fix it when we upgrading to use Scala 3

### Why are the changes needed?
This release improves collections, adds support for JDK 20 and 21, adds support for JDK 17 `sealed`:
- scala/scala#10363
- scala/scala#10184
- scala/scala#10397
- scala/scala#10348
- scala/scala#10105

There are 2 known issues in this version:

- scala/bug#12800
- scala/bug#12799

For the first one, there is no compilation warning messages related to `match may not be exhaustive` in Spark compile log, and for the second one, there is no case of `method.isAnnotationPresent(Deprecated.class)` in Spark code, there is just

https://github.com/apache/spark/blob/8c84d2c9349d7b607db949c2e114df781f23e438/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala#L130

in Spark Code, and I checked `javax.annotation.Nonnull` no this issue.

So I think These two issues will not affect Spark itself, but this doesn't mean it won't affect the code written by end users themselves

The full release notes as follows:

- https://github.com/scala/scala/releases/tag/v2.13.11

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
- Existing Test

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #42918 from LuciferYang/SPARK-40497-2.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Sep 16, 2023
1 parent 69ba1bd commit 55764a9
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 2 deletions.
6 changes: 5 additions & 1 deletion pom.xml
Expand Up @@ -3630,7 +3630,7 @@
<profile>
<id>scala-2.13</id>
<properties>
<scala.version>2.13.8</scala.version>
<scala.version>2.13.11</scala.version>
<scala.binary.version>2.13</scala.binary.version>
</properties>
<build>
Expand Down Expand Up @@ -3689,6 +3689,10 @@
-->
<arg>-Wconf:cat=unused-imports&amp;src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s</arg>
<arg>-Wconf:cat=unused-imports&amp;src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s</arg>
<!--
SPARK-40497 Upgrade Scala to 2.13.11 and suppress `Implicit definition should have explicit type`
-->
<arg>-Wconf:msg=Implicit definition should have explicit type:s</arg>
</args>
<compilerPlugins combine.self="override">
</compilerPlugins>
Expand Down
4 changes: 3 additions & 1 deletion project/SparkBuild.scala
Expand Up @@ -286,7 +286,9 @@ object SparkBuild extends PomBuild {
// TODO(SPARK-43850): Remove the following suppression rules and remove `import scala.language.higherKinds`
// from the corresponding files when Scala 2.12 is no longer supported.
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBase.scala:s",
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBaseOps.scala:s"
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBaseOps.scala:s",
// SPARK-40497 Upgrade Scala to 2.13.11 and suppress `Implicit definition should have explicit type`
"-Wconf:msg=Implicit definition should have explicit type:s"
)
}
}
Expand Down

0 comments on commit 55764a9

Please sign in to comment.