Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.5.0
-
None
-
None
-
Reviewed
Description
In the present implementation, the available buffer is divided into blocks (specified in the constructor for pipeline sort). and a linked list of these block byte buffers is maintained.
A span is created out of the buffers.
The present logic, doesnot handle scenario where a single key-value pair size doesnot fit into any of the blocks.
example if 1mb total memory is divided into 4 blocks, (256 kb each),
if a single KV pair is greater than the blocksize(~ignoring meta data size),
then it fails with buffer exceptions.
Attachments
Attachments
Issue Links
- is depended upon by
-
TEZ-2574 Make a better Metadata Value split choice in Pipeline sort
-
- Open
-