[SPARK-23322] Launcher handles can miss application updates if application finishes too quickly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 02/Feb/18 22:00
• Updated: 08/Oct/19 05:44
• Resolved: 08/Oct/19 05:44

Description

This is the underlying issue in SPARK-23020, which was worked around in our tests, but still exist in the code.

If a child application finishes too quickly, the launcher code may clean up the handle's state before the connection from the child has been acknowledged. This means than the application handle will have a final state LOST instead of whatever final state the application sent.

This doesn't seem to affect child processes as much as the new in-process launch mode, and it requires the child application to finish very quickly, which should be rare for the kind of use case the launcher library covers.

Issue Links

• relates to: SPARK-23020 Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher