[MAPREDUCE-7052] TestFixedLengthInputFormat#testFormatCompressedIn is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
• Component/s: client, test
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 13/Feb/18 12:13
• Updated: 15/Feb/18 23:18
• Resolved: 15/Feb/18 21:44

Description

Sometimes the test case TestFixedLengthInputFormat#testFormatCompressedIn can fail with the following error:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:322)
	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90)

Root cause: under special circumstances, the following line can return a huge number:

          // Test a split size that is less than record len
          numSplits = (int)(fileSize/Math.floor(recordLength/2));

For example, let seed be 2026428718. This causes recordLength to be 1 at iteration 19. Math.floor() returns negative Infinity, which becomes positve infinity after the divison. Casting it to int yields Integer.MAX_VALUE. Eventually we get an OOME because the test wants to create a huge InputSplit array.