[SPARK-2425] Standalone Master is too aggressive in removing Applications

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Mark Hamstra
• Reporter: Mark Hamstra
• Created: 09/Jul/14 23:49
• Updated: 09/Sep/14 18:18
• Resolved: 09/Sep/14 18:17

Description

When standalone Executors trying to run a particular Application fail a cummulative ApplicationState.MAX_NUM_RETRY times, Master will remove the Application. This will be true even if there actually are a number of Executors that are successfully running the Application. This makes long-running standalone-mode Applications in particular unnecessarily vulnerable to limited failures in the cluster – e.g., a single bad node on which Executors repeatedly fail for any reason can prevent an Application from starting or can result in a running Application being removed even though it could continue to run successfully (just not making use of all potential Workers and Executors.)

Issue Links

• is related to: SPARK-2424 ApplicationState.MAX_NUM_RETRY should be configurable
• relates to: SPARK-3289 Avoid job failures due to rescheduling of failing tasks on buggy machines
• links to: [Github] Pull Request #1360 (markhamstra)