[SPARK-2140] yarn stable client doesn't properly handle MEMORY_OVERHEAD for AM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: YARN
• Target Version/s: 1.1.0
• Assignee: Chris Cope
• Reporter: Thomas Graves
• Created: 13/Jun/14 15:16
• Updated: 11/Sep/14 13:14
• Resolved: 11/Sep/14 13:14

Description

The yarn stable client doesn't properly remove the MEMORY_OVERHEAD amount from the java heap size, the code to handle that is commented out (see function calculateAMMemory). We should fix this.

Issue Links

• is duplicated by: SPARK-1287 yarn alpha and stable Client calculateAMMemory routines are different
• is related to: SPARK-2604 Spark Application hangs on yarn in edge case scenario of executor memory requirement
• links to: [Github] Pull Request #1906 (copester)
• links to: Web Link [Github] Pull Request #1906 (copester)