[SPARK-22199] Spark Job on YARN fails with executors "Slave registration failed"

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 1.6.3
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Prabhu Joseph
• Created: 04/Oct/17 11:47
• Updated: 17/May/20 18:14
• Resolved: 12/Feb/19 21:23

Description

Spark Job on YARN Failed with max executors Failed.

ApplicationMaster logs:

17/09/28 04:18:27 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Max number of executor failures (3) reached)

Checking the failed container logs shows "Slave registration failed: Duplicate executor ID" whereas the Driver logs shows it has removed those executors as they are idle for spark.dynamicAllocation.executorIdleTimeout

Executor Logs:

17/09/28 04:18:26 ERROR CoarseGrainedExecutorBackend: Slave registration failed: Duplicate executor ID: 122

Driver logs:

17/09/28 04:18:21 INFO ExecutorAllocationManager: Removing executor 122 because it has been idle for 60 seconds (new desired total will be 133)

There are two issues here:

1. Error Message in executor is misleading "Slave registration failed: Duplicate executor ID" as the actual error is it was idle
2. The job failed as there are executors idle for spark.dynamicAllocation.executorIdleTimeout