Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7052

TestFixedLengthInputFormat#testFormatCompressedIn is flaky

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Sometimes the test case TestFixedLengthInputFormat#testFormatCompressedIn can fail with the following error:

      java.lang.OutOfMemoryError: Requested array size exceeds VM limit
      	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:322)
      	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90)
      

      Root cause: under special circumstances, the following line can return a huge number:

                // Test a split size that is less than record len
                numSplits = (int)(fileSize/Math.floor(recordLength/2));
      

      For example, let seed be 2026428718. This causes recordLength to be 1 at iteration 19. Math.floor() returns negative Infinity, which becomes positve infinity after the divison. Casting it to int yields Integer.MAX_VALUE. Eventually we get an OOME because the test wants to create a huge InputSplit array.

        Attachments

        1. MAPREDUCE-7052-001.patch
          1 kB
          Peter Bacsko
        2. MAPREDUCE-7052-002.patch
          1 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              pbacsko Peter Bacsko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: