[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

dongjoon-hyun · 2021-05-28T16:29:37Z

What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.

[WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 #27929 (Upgrade Scala to 2.12.11, @wangyum )
[SPARK-33921][BUILD] Upgrade Scala version to 2.12.12 #30940 (Upgrade Scala to 2.12.12, @viirya )
[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 #31223 (Upgrade Scala to 2.12.13, @dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.

Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:

silencer library is released via Add 2.12.14 ghik/silencer#66
genjavadoc library is released via Release for Scala 2.12.14 lightbend/genjavadoc#282

Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.

Does this PR introduce any user-facing change?

Yes, but this is a bug-fixed version.

How was this patch tested?

Pass the CIs.

SparkQA · 2021-05-28T16:41:25Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43586/

SparkQA · 2021-05-28T16:44:44Z

Test build #139065 has finished for PR 32697 at commit 274de13.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2021-05-29T14:10:49Z

Looks good if it passes, but looks like some library is missing:

sbt.librarymanagement.ResolveException: Error downloading com.github.ghik:silencer-lib_2.12.14:1.7.5
[error] Not found
[error] Not found
[error] not found: https://maven-central.storage-download.googleapis.com/maven2/com/github/ghik/silencer-lib_2.12.14/1.7.5/silencer-lib_2.12.14-1.7.5.pom
[error] not found: https://repo1.maven.org/maven2/com/github/ghik/silencer-lib_2.12.14/1.7.5/silencer-lib_2.12.14-1.7.5.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/2/.m2/repository/com/github/ghik/silencer-lib_2.12.14/1.7.5/silencer-lib_2.12.14-1.7.5.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/2/.ivy2/localcom.github.ghik/silencer-lib_2.12.14/1.7.5/ivys/ivy.xml

iemejia · 2021-05-29T17:57:38Z

From https://github.com/ghik/silencer:

NOTE: Scala 2.13.2 and 2.12.13 introduced configurable warnings.
This means that unless you're still cross compiling for Scala 2.11, this plugin is obsolete, and you should use
@nowarn.

Maybe we can remove it and avoid this error.

srowen · 2021-05-29T19:26:31Z

I see, just exclude it? possibly. Looks like it may just need some time to get cross-published as the artifact appears quite specific to the scala version: https://search.maven.org/search?q=silencer-lib

dongjoon-hyun · 2021-05-29T22:16:18Z

Hi, @srowen and @iemejia . Yes, it does.

As I wrote in the PR description, I made a PR to silencer and it's merged three hours ago and released.

silencer library is not released yet. ghik/silencer#66

This is the released one.

https://repo1.maven.org/maven2/com/github/ghik/silencer-plugin_2.12.14/1.7.5/

I'll rebase this PR to the branch to retrigger it. However, the second item in the PR description, genjavadoc is still not ready as I mentioned also.

genjavadoc library is not released yet. https://repo1.maven.org/maven2/com/typesafe/genjavadoc/

And, @iemejia . I'm intentionally not going down that path. :) We can do that separately after this PR.

SparkQA · 2021-05-29T22:49:11Z

Test build #139082 has finished for PR 32697 at commit 0921ed2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-29T23:25:05Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43603/

dongjoon-hyun · 2021-05-30T20:48:17Z

Now, genjavadoc is ready.

Release for Scala 2.12.14 lightbend/genjavadoc#282

SparkQA · 2021-05-30T21:29:14Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43606/

viirya

Looks good if CI can pass.

dongjoon-hyun · 2021-05-30T21:43:54Z

Thank you, @viirya !

dongjoon-hyun · 2021-05-30T23:07:49Z

Thank you, @srowen , @iemejia , @viirya .

GitHub Action passed
Jenkins Scala/Java passed and currently SparkR test is running.

I'll merge this PR for Apache Spark 3.2.0.

BTW, cc @Ngone51 since he reported the cyclic dependency issue last time.

SparkQA · 2021-05-30T23:13:49Z

Test build #139085 has finished for PR 32697 at commit 493b8be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zzcclp · 2021-05-31T02:45:02Z

pom.xml

@@ -162,7 +162,7 @@
    <commons.math3.version>3.4.1</commons.math3.version>
    <!-- managed up from 3.2.1 for SPARK-11652 -->
    <commons.collections.version>3.2.2</commons.collections.version>
-    <scala.version>2.12.10</scala.version>
+    <scala.version>2.12.14</scala.version>


@dongjoon-hyun please modify the scala version to 2.12.14 in profile scala-2.12, otherwise there maybe a Error:scala: bad option: -P:silencer:globalFilters=.*deprecated.* error when tick the profile scala-2.12.

Oh, thanks. I missed there.

### What changes were proposed in this pull request? This PR is a follow-up of #32697 to update the missed part. After SPARK-34774, we have Scala 2.12 version in `scala-2.12` profile. ### Why are the changes needed? To be consistent. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual. **BEFORE** ``` $ build/mvn help:evaluate -Pscala-2.12 -Dexpression=scala.version | grep "^2.12" Using `mvn` from path: /usr/local/bin/mvn 2.12.10 ``` **AFTER** ``` $ build/mvn help:evaluate -Pscala-2.12 -Dexpression=scala.version | grep "^2.12" Using `mvn` from path: /usr/local/bin/mvn 2.12.14 ``` Closes #32707 from dongjoon-hyun/SPARK-31168-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

iemejia · 2021-05-31T06:46:07Z

Excellent @dongjoon-hyun! My suggestion to remove silencer was to unblock the issue but this can be definitely be done afterwards.
Great to see the new version of Scala just for the support for more recent JVMs available, hopefully this will win us time to be ready when JVM 17 is released!

dongjoon-hyun · 2021-05-31T06:49:11Z

Sure, I agree with you that we need to remove it in Apache Spark 3.2.0 timeframe. Thanks, @iemejia .

My suggestion to remove silencer was to unblock the issue but this can be definitely be done afterwards.

* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-36759][BUILD] Upgrade Scala to 2.12.15 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better. ### Why are the changes needed? Scala 2.12.15 improves compatibility with JDK 17 and 18: https://github.com/scala/scala/releases/tag/v2.12.15 - Avoids IllegalArgumentException in JDK 17+ for lambda deserialization - Upgrades to ASM 9.2, for JDK 18 support in optimizer ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass the CIs Closes #33999 from dongjoon-hyun/SPARK-36759. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 16f1f71) * [SPARK-36759][BUILD][FOLLOWUP] Update version in scala-2.12 profile and doc ### What changes were proposed in this pull request? This is a follow-up to fix the leftover during switching the Scala version. ### Why are the changes needed? This should be consistent. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is not tested by UT. We need to check manually. There is no more `2.12.14`. ``` $ git grep 2.12.14 R/pkg/tests/fulltests/test_sparkSQL.R: c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16"))) data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799 sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying ``` Closes #34020 from dongjoon-hyun/SPARK-36759-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit adbea25) * [SPARK-39414][BUILD] Upgrade Scala to 2.12.16 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.16 ### Why are the changes needed? This version bring some bug fix and start to try to support Java 19 scala/scala@v2.12.15...v2.12.16 - [Upgrade to asm 9.3, for JDK19 support](scala/scala#10000) - [Fix codegen for MH.invoke etc under JDK 17 -release](scala/scala#9930) - [Deprecation related SecurityManager on JDK 17 ](scala/scala#9775) ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass Github Actions Closes #36807 from LuciferYang/SPARK-39414. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ed875a8) * fix * fix * fix * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/core/src/main/scala/org/apache/spark/scheduler/SpillableTaskResultGetter.scala:36: non-variable type argument org.apache.spark.scheduler.DirectTaskResult[_] in type pattern scala.runtime.NonLocalReturnControl[org.apache.spark.scheduler.DirectTaskResult[_]] is unchecked since it is eliminated by erasure�[0m �[0m[�[31merror�[0m] �[0m[warn] private[spark] class SpillableTaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedulerImpl)�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:287: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "^")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] private val pow: Parser[Term] = term ~ "^" ~ "^[1-9]\\d*".r ^^ {�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:301: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "~")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] (label ~ "~" ~ expr) ^^ { case r ~ "~" ~ t => ParsedRFormula(r, t.asTerms.terms) }�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: yangjie01 <yangjie01@baidu.com>

github-actions bot added BUILD DOCS labels May 28, 2021

dongjoon-hyun marked this pull request as draft May 28, 2021 16:31

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14

0921ed2

empty commit

493b8be

dongjoon-hyun marked this pull request as ready for review May 30, 2021 21:04

viirya reviewed May 30, 2021

View reviewed changes

dongjoon-hyun closed this in 6c4b60f May 30, 2021

viirya approved these changes May 30, 2021

View reviewed changes

dongjoon-hyun deleted the SPARK-31168 branch May 30, 2021 23:09

zzcclp reviewed May 31, 2021

View reviewed changes

dongjoon-hyun mentioned this pull request May 31, 2021

[SPARK-31168][BUILD][FOLLOWUP] Update scala-2.12 profile #32707

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

dongjoon-hyun commented May 28, 2021 •

edited

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

srowen commented May 29, 2021

iemejia commented May 29, 2021

srowen commented May 29, 2021

dongjoon-hyun commented May 29, 2021 •

edited

SparkQA commented May 29, 2021

SparkQA commented May 29, 2021

dongjoon-hyun commented May 30, 2021

SparkQA commented May 30, 2021

viirya left a comment

dongjoon-hyun commented May 30, 2021

dongjoon-hyun commented May 30, 2021 •

edited

SparkQA commented May 30, 2021

zzcclp May 31, 2021

dongjoon-hyun May 31, 2021

iemejia commented May 31, 2021

dongjoon-hyun commented May 31, 2021

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

Conversation

dongjoon-hyun commented May 28, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

srowen commented May 29, 2021

iemejia commented May 29, 2021

srowen commented May 29, 2021

dongjoon-hyun commented May 29, 2021 • edited

SparkQA commented May 29, 2021

SparkQA commented May 29, 2021

dongjoon-hyun commented May 30, 2021

SparkQA commented May 30, 2021

viirya left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented May 30, 2021

dongjoon-hyun commented May 30, 2021 • edited

SparkQA commented May 30, 2021

zzcclp May 31, 2021

Choose a reason for hiding this comment

dongjoon-hyun May 31, 2021

Choose a reason for hiding this comment

iemejia commented May 31, 2021

dongjoon-hyun commented May 31, 2021

dongjoon-hyun commented May 28, 2021 •

edited

dongjoon-hyun commented May 29, 2021 •

edited

dongjoon-hyun commented May 30, 2021 •

edited