The PostCommit Java PVR Spark Batch job is flaky

[github-actions]

The PostCommit Java PVR Spark Batch is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Spark_Batch.yml?query=is%3Afailure+branch%3Amaster to see the logs.

[Abacn]

Random tests in this test suite failing due to tmp file get deleted half way, likely a racing issue.

This is recurring for a long time.

java.lang.RuntimeException: The Runner experienced the following error during execution:
java.lang.RuntimeException: Error received from SDK harness for instruction 4: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
	at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:78)
	at org.apache.beam.sdk.util.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:767)
	at org.apache.beam.sdk.util.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:391)
	at org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:379)
	...
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.InvalidClassException: org.apache.beam.sdk.transforms.ParDoTest$StateTests; unable to create instance
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2200)
	...
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:75)
	... 19 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedSerializationConstructorAccessor390.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1102)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2197)
	... 31 more
Caused by: java.util.ServiceConfigurationError: org.apache.beam.sdk.io.FileSystemRegistrar: Error reading configuration file
	...
	at org.apache.beam.sdk.transforms.ParDoTest$SharedTestBase.<init>(ParDoTest.java:193)
	... 35 more
Caused by: java.io.FileNotFoundException: /tmp/spark-4426f648-ee8b-49e3-9545-dffde22d1a6f/userFiles-b02ee903-820b-4b77-b1b7-69f020d54c8b/beam-sdks-java-extensions-google-cloud-platform-core-2.57.0-SNAPSHOT.jar (No such file or directory)
	at java.util.zip.ZipFile.open(Native Method)
	...
	at java.net.URL.openStream(URL.java:1093)
	at java.util.ServiceLoader.parse(ServiceLoader.java:304)
	... 46 more

	at org.apache.beam.runners.portability.JobServicePipelineResult.propagateErrors(JobServicePipelineResult.java:176)
	at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:117)
	at org.apache.beam.runners.portability.testing.TestPortableRunner.run(TestPortableRunner.java:81)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:404)
	...
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)