[TEZ-3634] reduce the buffer sizes in PipelinedSorter by a small amount

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Target Version/s: 0.8.5
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 22/Feb/17 05:21
• Updated: 22/Aug/17 00:02
• Resolved: 23/Feb/17 01:11

Description

When running in large JVMs with the G1GC, the regionsize maxes out at 32MB.

PipelinedSorter does allocations of 32MB and 256MB. The actual allocation ends up being 24 bytes over 32M/256M. As a result the 32M allocation ends up taking 2 regions (or 64M), and the 256M ends up taking 9 regions instead of 8. Also leads to fragmentation of the heap, resulting in more Full GCs than required.

Live humongous region 2 size 33554456 start 0x00007f4918000000 length 2 with remset 0 code roots 0 is marked 0 reclaim candidate 0 type array 1

Dead humongous region 111 size 268435480 start 0x00007f49f2000000 length 9 with remset 0 code roots 0 is marked 0 reclaim candidate 1 type array 1