[HADOOP-6766] Spill can fail with bad call to Random

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 0.20.2
• Component/s: fs
• Assignee: Unassigned
• Reporter: Peter Arthur Ciccolo
• Created: 14/May/10 22:43
• Updated: 25/Sep/15 17:10
• Resolved: 22/Jul/14 19:30

Description

java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:250)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:243)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:289)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:159)

confChanged assumes that the list of dirs it creates (LocalDirAllocator.java:215) has at least one element in it by the end of the function. If, for each local dir, either the conditional on line 221 is false, or the call to DiskChecker.checkDir() throws an exception, this assumption will not hold. In this case, dirIndexRandomizer.nextInt() is called on the number of elements in dirs, which is 0. Since dirIndexRandomizer (195) is an instance of Random(), it needs a positive (non-zero) argument to nextInt().

Issue Links

• duplicates: MAPREDUCE-635 IllegalArgumentException is thrown if mapred local dir is not writable.
• is related to: HADOOP-8437 getLocalPathForWrite should throw IOException for invalid paths