[Python][CI] test-conda-python-3.11-spark-master fails, requires Java 17

[danepitkin]

Describe the bug, including details regarding any error messages, version, and platform.

test-conda-python-3.11-spark-master fails with:

Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/com/github/os72/protoc-jar-maven-plugin/3.11.4/protoc-jar-maven-plugin-3.11.4.jar (10 MB at 11 MB/s)
Using `mvn` from path: /spark/build/apache-maven-3.9.4/bin/mvn
Error:  Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.3.0:enforce (enforce-versions) on project spark-parent_2.13: 
Error:  Rule 1: org.apache.maven.enforcer.rules.version.RequireJavaVersion failed with message:
Error:  Detected JDK version 1.8.0-382 (JAVA_HOME=/opt/conda/envs/arrow/jre) is not in the allowed range [17,).
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
1
Error: `docker-compose --file /home/runner/work/crossbow/crossbow/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=14.0.0.dev256 conda-python-spark` exited with a non-zero exit code 1, see the process log above.

https://github.com/ursacomputing/crossbow/actions/runs/6424410884/job/17445002795

Component(s)

Python

[lidavidm]

I suppose if Spark has dropped Java 8, there's no reason we can't drop Java 8 for Arrow 14 or Arrow 15?

[danepitkin]

+1! I just opened #38051 today.

Arrow v15 is the current target release. I figured it would be better to have more discussion on the mailing list before dropping support.