[HDFS-2541] For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1
• Fix Version/s: 0.23.1, 1.1.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Harsh J
• Reporter: Harsh J
• Created: 05/Nov/11 08:38
• Updated: 19/Mar/13 17:59
• Resolved: 20/Nov/11 03:04

Description

Running off 0.20-security, I noticed that one could get the following exception when scanners are used:

DataXceiver 
java.lang.IllegalArgumentException: n must be positive 
at java.util.Random.nextInt(Random.java:250) 
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.getNewBlockScanTime(DataBlockScanner.java:251) 
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.addBlock(DataBlockScanner.java:268) 
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:432) 
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)

This is cause the period, determined in the DataBlockScanner (0.20+) or BlockPoolSliceScanner (0.23+), is cast to an integer before its sent to a Random.nextInt(...) call. For sufficiently large values of the long 'period', the casted integer may be negative. This is not accounted for. I'll attach a sample test that shows this possibility with the numbers.

We should ensure we do a Math.abs(...) before we send it to the Random.nextInt(...) call to avoid this.

With this bug, the maximum # of blocks a scanner may hold in its blocksMap without opening up the chance for beginning this exception (intermittent, as blocks continue to grow) would be 3582718.

Issue Links

• is related to: HDFS-3488 BlockPoolSliceScanner#getNewBlockScanTime does not handle numbers > 31 bits properly
• relates to: HDFS-3456 blockReportInterval is long value but when we take the random value it uses getRandom().nextInt,it is causing frequently BR