[SPARK-1287] yarn alpha and stable Client calculateAMMemory routines are different

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.9.0, 1.0.0
• Component/s: YARN
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 20/Mar/14 08:51
• Updated: 22/Aug/14 21:50
• Resolved: 22/Aug/14 21:50

Description

The yarn alpha version of calculateAMMemory takes into account the minimum resource capability and also subtracts out YarnAllocationHandler.MEMORY_OVERHEAD.

The yarn stable version just sets the -Xmx to whatever is passed in by the user.

The 2 of these should be the same.

Personally I also think its weird how spark currently takes whatever user passes in for memory and adds YarnAllocationHandler.MEMORY_OVERHEAD to request to RM. This can be confusing to users. We should revisit all of this and commonize the stable/alpha code where possible.

Issue Links

• duplicates: SPARK-2140 yarn stable client doesn't properly handle MEMORY_OVERHEAD for AM