[MAPREDUCE-6549] multibyte delimiters with LineRecordReader cause duplicate records

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
• Component/s: mrv1, mrv2
• Hadoop Flags: Reviewed
• Assignee: Wilfred Spiegelenburg
• Reporter: Dustin Cote
• Created: 15/Nov/15 00:32
• Updated: 26/Feb/20 08:50
• Resolved: 26/Nov/15 04:06

Description

LineRecorderReader currently produces duplicate records under certain scenarios such as:

1) input string: "abc++defghi+"
delimiter string: "+++"
test passes with all sizes of the split
2) input string: "abc+def+ghi+"
delimiter string: "+++"
test fails with a split size of 4
2) input string: "abc++defghi+"
delimiter string: "++"
test fails with a split size of 5
3) input string "abc++defghij+"
delimiter string: "++"
test fails with a split size of 4
4) input string "abc+def+ghi+"
delimiter string: "++"
test fails with a split size of 9

Issue Links

• is duplicated by: MAPREDUCE-6891 TextInputFormat: duplicate records with custom delimiter
• relates to: MAPREDUCE-6481 LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.
• relates to: MAPREDUCE-6558 multibyte delimiters with compressed input files generate duplicate records