Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6766

Spill can fail with bad call to Random

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      java.lang.IllegalArgumentException: n must be positive
      at java.util.Random.nextInt(Random.java:250)
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:243)
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:289)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
      at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
      at org.apache.hadoop.mapred.Child.main(Child.java:159)

      confChanged assumes that the list of dirs it creates (LocalDirAllocator.java:215) has at least one element in it by the end of the function. If, for each local dir, either the conditional on line 221 is false, or the call to DiskChecker.checkDir() throws an exception, this assumption will not hold. In this case, dirIndexRandomizer.nextInt() is called on the number of elements in dirs, which is 0. Since dirIndexRandomizer (195) is an instance of Random(), it needs a positive (non-zero) argument to nextInt().

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ciccolo Peter Arthur Ciccolo
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: