[TEZ-2574] Make a better Metadata Value split choice in Pipeline sort

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Saikat
• Reporter: Saikat
• Created: 24/Jun/15 15:22
• Updated: 08/Sep/16 21:05

Description

In the current implementation of pipeline sort, when a new sort span object is created with a hard coded value of 1M items and 16 bytes per item.
According to the present code logic,
int metasize = METASIZE*maxItems;
int dataSize = maxItems * perItem;
if(capacity < (metasize+dataSize))

{ // try to allocate less meta space, because we have sample data metasize = METASIZE*(capacity/(perItem+METASIZE)); }

if capacity is less than 32mb, the buffer will be halved into meta and value buffers, which is not efficient.
We need a more generic split, based on the KV pair size written to the buffer.

Issue Links

• depends upon: TEZ-2575 Handle KeyValue pairs size which do not fit in a single block in PipelinedSorter
• depends upon: TEZ-2643 Minimize number of empty spills in Pipelined Sorter