[HADOOP-13270] BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Target Version/s: 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Kai
• Reporter: Haibo Chen
• Created: 24/Feb/16 01:41
• Updated: 04/Oct/17 20:26
• Resolved: 14/Jun/16 01:29

Description

Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is 1313094493.

Stacktrace
java.lang.AssertionError: Key in multiple partitions.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)

Issue Links

• breaks: HADOOP-14919 BZip2 drops records when reading data in splits
• is broken by: HADOOP-11445 Bzip2Codec: Data block is skipped when position of newly created stream is equal to start of split