[AMBARI-17818] Tez & Hive: Memory management should use scaled down Xmx for sizing buffers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: stacks
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 20/Jul/16 17:43
• Updated: 22/Jul/16 00:50
• Resolved: 21/Jul/16 21:42

Description

The defaults in Ambari assume 80% of the YARN container size is allocate to the user heap (Xmx).

On top of this particular restriction, the allocation blocks are divided into Eden/Survivor/Tenured - a single allocation spanning 40% of the heap might fail for tez runtime.io.sort.mb.

Scale down the Tez sort size and map join by 0.8x to match Xmx.