[SPARK-12755] Spark may attempt to rebuild application UI before finishing writing the event logs in possible race condition

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.2
• Fix Version/s: 1.5.3, 1.6.1, 2.0.0
• Component/s: Spark Core
• Assignee: Michael MacFadden
• Reporter: Michael MacFadden
• Created: 11/Jan/16 18:59
• Updated: 13/May/16 06:08
• Resolved: 25/Jan/16 09:52

Description

As reported in SPARK-6950, it appears that sometimes the standalone master attempts to build an application's historical UI before closing the app's event log. This is still an issue for us in 1.5.2+, and I believe I've found the underlying cause.

When stopping a SparkContext, the stop method stops the DAG scheduler:

https://github.com/apache/spark/blob/a76cf51ed91d99c88f301ec85f3cda1288bcf346/core/src/main/scala/org/apache/spark/SparkContext.scala#L1722-L1727

and then stops the event logger:

https://github.com/apache/spark/blob/a76cf51ed91d99c88f301ec85f3cda1288bcf346/core/src/main/scala/org/apache/spark/SparkContext.scala#L1722-L1727

Though it is difficult to follow the chain of events, one of the sequelae of stopping the DAG scheduler is that the master's rebuildSparkUI method is called. This method looks for the application's event logs, and its behavior varies based on the existence of an .inprogress file suffix. In particular, a warning is logged if this suffix exists:

https://github.com/apache/spark/blob/a76cf51ed91d99c88f301ec85f3cda1288bcf346/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L935

After calling the stop method on the DAG scheduler, the SparkContext stops the event logger:

https://github.com/apache/spark/blob/a76cf51ed91d99c88f301ec85f3cda1288bcf346/core/src/main/scala/org/apache/spark/SparkContext.scala#L1734-L1736

This renames the event log, dropping the .inprogress file sequence.

As such, a race condition exists where the master may attempt to process the application log file before finalizing it.

Issue Links

• is related to: SPARK-6950 Spark master UI believes some applications are in progress when they are actually completed
• links to: [Github] Pull Request #10700 (mallman)