[BEAM-8867] sparkValidatesRunner - python post commit flaky

• Type: Sub-task
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: runner-spark
• Labels: flake
• Assignee: Kyle Weaver
• Reporter: Ahmet Altay
• Created: 02/Dec/19 20:55
• Updated: 22/Aug/20 15:12
• Resolved: 12/Dec/19 17:21

Description

Logs: https://scans.gradle.com/s/iwdvlnwg6euhk/console-log?task=:sdks:python:test-suites:portable:py2:sparkValidatesRunner

 

Error:

ERROR org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_79e29409-03c8-4203-bac4-ca3c30f00b35","basePath":"/tmp/sparktestzSz0RX"}: {} ERROR org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_79e29409-03c8-4203-bac4-ca3c30f00b35","basePath":"/tmp/sparktestzSz0RX"}: {} java.io.FileNotFoundException: /tmp/sparktestzSz0RX/job_79e29409-03c8-4203-bac4-ca3c30f00b35/MANIFEST (No such file or directory)

Issue Links

• duplicates: BEAM-8870 beam_PostCommit_Python_VR_Spark is permanently failing
• relates to: BEAM-8883 downgrade "Failed to remove job staging directory" log level