[SPARK-40497][BUILD] Re-upgrade Scala to 2.13.11

[LuciferYang]

What changes were proposed in this pull request?

This PR aims to re-upgrade Scala to 2.13.11, after SPARK-45144 was merged, the build issues mentioned in #41943 should no longer exist.

• https://www.scala-lang.org/news/2.13.11

Additionally, this pr adds a new suppression rule for warning message: Implicit definition should have explicit type, this is a new compile check introduced by scala/scala#10083, we must fix it when we upgrading to use Scala 3

Why are the changes needed?

This release improves collections, adds support for JDK 20 and 21, adds support for JDK 17 sealed:

• Support JDK 21 by upgrading to ASM 9.5 scala/scala#10363
• Support JDK 20 by upgrading to ASM 9.4 scala/scala#10184
• ClassfileParser: allow missing param names (for JDK 21) scala/scala#10397
• Support Java 17 sealed in Java sources scala/scala#10348
• Support JDK 17 PermittedSubclasses in classfiles scala/scala#10105

There are 2 known issues in this version:

• Exhaustivity checker emits spurious warning when matching on Java enum type (2.13.11 regression) scala/bug#12800
• Duplicated @Deprecated annotations when extending Java interface with deprecated default method cause java.lang.annotation.AnnotationFormatError when accessed via Java reflection (2.13.11 regression) scala/bug#12799

For the first one, there is no compilation warning messages related to match may not be exhaustive in Spark compile log, and for the second one, there is no case of method.isAnnotationPresent(Deprecated.class) in Spark code, there is just

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala

Line 130 in 8c84d2c

val hasNonNull = readMethod.isAnnotationPresent(classOf[Nonnull])

in Spark Code, and I checked @javax.annotation.Nonnull no this issue.

So I think These two issues will not affect Spark itself, but this doesn't mean it won't affect the code written by end users themselves

The full release notes as follows:

• https://github.com/scala/scala/releases/tag/v2.13.11

Does this PR introduce any user-facing change?

Yes, this is a Scala version change.

How was this patch tested?

• Existing Test

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

will revert after test

[LuciferYang]

https://github.com/LuciferYang/spark/runs/16776579478
https://github.com/LuciferYang/spark/runs/16800035858

Tested Scala 2.13 + Java 17, onnly the org.apache.spark.sql.connect.execution.ReattachableExecuteSuite test failed, which is a known issue.

[LuciferYang]

cc @dongjoon-hyun FYI
also cc @eejbyfeldt for double check

[eejbyfeldt]

Test building 2.13 with both Java 11 and Java 17 worked like expected. Thanks for fixing this!

2.13.12 has also been release https://github.com/scala/scala/releases/tag/v2.13.12 so we might want to look into that in the future.

[LuciferYang]

Thanks @dongjoon-hyun and @eejbyfeldt

[LuciferYang]

2.13.12 has also been release https://github.com/scala/scala/releases/tag/v2.13.12 so we might want to look into that in the future.

We can test it once Ammonite releases a new version supporting Scala 2.13.12.

[dongjoon-hyun]

Could you re-trigger the failed pipelines?

[LuciferYang]

Could you re-trigger the failed pipelines?

Triggered

[dongjoon-hyun]

Merged to master for Apache Spark 4.0.0. Thank you, @LuciferYang and all.