[MAPREDUCE-6558] multibyte delimiters with compressed input files generate duplicate records

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
• Component/s: mrv1, mrv2
• Target Version/s: 2.8.0, 2.7.3, 2.6.5
• Hadoop Flags: Reviewed
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 24/Nov/15 10:41
• Updated: 26/Feb/20 05:51
• Resolved: 13/May/16 14:40

Description

This is the follow up for MAPREDUCE-6549. Compressed files cause record duplications as shown in different junit tests. The number of duplicated records changes with the splitsize:

Unexpected number of records in split (splitsize = 10)
Expected: 41051
Actual: 45062

Unexpected number of records in split (splitsize = 100000)
Expected: 41051
Actual: 41052

Test passes with splitsize = 147445 which is the compressed file length.The file is a bzip2 file with 100k blocks and a total of 11 blocks

Issue Links

• is related to: MAPREDUCE-6549 multibyte delimiters with LineRecordReader cause duplicate records