[TEZ-2575] Handle KeyValue pairs size which do not fit in a single block in PipelinedSorter

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.8.0-alpha, 0.7.1
• Hadoop Flags: Reviewed
• Assignee: Saikat
• Reporter: Saikat
• Created: 24/Jun/15 15:38
• Updated: 02/Sep/15 14:24
• Resolved: 04/Jul/15 11:55

Description

In the present implementation, the available buffer is divided into blocks (specified in the constructor for pipeline sort). and a linked list of these block byte buffers is maintained.
A span is created out of the buffers.
The present logic, doesnot handle scenario where a single key-value pair size doesnot fit into any of the blocks.
example if 1mb total memory is divided into 4 blocks, (256 kb each),
if a single KV pair is greater than the blocksize(~ignoring meta data size),
then it fails with buffer exceptions.

Issue Links

• is depended upon by: TEZ-2574 Make a better Metadata Value split choice in Pipeline sort