[BEAM-9971] beam_PostCommit_Java_PVR_Spark_Batch flakes (no such file)

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.22.0
• Component/s: test-failures
• Labels: portability-spark
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 12/May/20 19:05
• Updated: 28/May/20 17:50
• Resolved: 27/May/20 23:00

Description

This happens sporadically. One time the issue affected 14 tests; another time it affected 112 tests.

It looks like the ClassLoader is sometimes contaminated with jars from /tmp/spark-*, which have already been deleted.

20/05/21 13:54:27 ERROR org.apache.beam.runners.jobsubmission.JobInvocation: Error during job invocation pipelinetest0testidentitytransform-kcweaver-0521205426-f4de06c4_51aced77-c171-4842-be1f-6c79226872e5.
java.util.ServiceConfigurationError: org.apache.beam.runners.core.construction.NativeTransforms$IsNativeTransform: Error reading configuration file
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.parse(ServiceLoader.java:309)
at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at org.apache.beam.runners.core.construction.NativeTransforms.isNative(NativeTransforms.java:50)
at org.apache.beam.runners.core.construction.graph.QueryablePipeline.isPrimitiveTransform(QueryablePipeline.java:189)
at org.apache.beam.runners.core.construction.graph.QueryablePipeline.getPrimitiveTransformIds(QueryablePipeline.java:137)
at org.apache.beam.runners.core.construction.graph.QueryablePipeline.forPrimitivesIn(QueryablePipeline.java:90)
at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.<init>(GreedyPipelineFuser.java:67)
at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:90)
at org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:94)
at org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:83)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /tmp/spark-5e8a8a9a-22d6-48d5-b398-1a4f5582d954/userFiles-ec74cac1-21b5-4127-b764-540636b733d0/beam-runners-core-construction-java-2.22.0-SNAPSHOT-tests.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:230)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.jar.JarFile.<init>(JarFile.java:167)
at java.util.jar.JarFile.<init>(JarFile.java:104)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
at java.net.URL.openStream(URL.java:1045)
at java.util.ServiceLoader.parse(ServiceLoader.java:304)
... 18 more

Issue Links

• is caused by: BEAM-10109 Fix context classloader in Spark portable runner
• links to: GitHub Pull Request #11784
• links to: GitHub Pull Request #11842