[SPARK-2604] Spark Application hangs on yarn in edge case scenario of executor memory requirement

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Twinkle Sachdeva
• Created: 21/Jul/14 10:37
• Updated: 15/Dec/14 05:28
• Resolved: 15/Dec/14 05:28

Description

In yarn environment, let's say :
MaxAM = Maximum allocatable memory
ExecMem - Executor's memory

if (MaxAM > ExecMem && ( MaxAM - ExecMem) > 384m ))
then Maximum resource validation fails w.r.t executor memory , and application master gets launched, but when resource is allocated and again validated, they are returned and application appears to be hanged.

Typical use case is to ask for executor memory = maximum allowed memory as per yarn config

Issue Links

• relates to: SPARK-2140 yarn stable client doesn't properly handle MEMORY_OVERHEAD for AM
• links to: [Github] Pull Request #1571 (twinkle-sachdeva)