[TEZ-3809] The buffer size allocated for InMemoryMapOutput can be optimized

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Muhammad Samir Khan
• Reporter: Muhammad Samir Khan
• Created: 01/Aug/17 21:20
• Updated: 07/Aug/17 19:07

Description

Related jiras: TEZ-3752 and TEZ-3732.

-When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732).

-Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end.

-InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This creates an inconsistency between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers.

-InMemoryWriter does not account for two EOF_MARKERs written at close() in its accounting so that the getRawLength() method is off by two bytes.