Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2541

For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.23.1, 1.1.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running off 0.20-security, I noticed that one could get the following exception when scanners are used:

      DataXceiver 
      java.lang.IllegalArgumentException: n must be positive 
      at java.util.Random.nextInt(Random.java:250) 
      at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.getNewBlockScanTime(DataBlockScanner.java:251) 
      at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.addBlock(DataBlockScanner.java:268) 
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:432) 
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
      

      This is cause the period, determined in the DataBlockScanner (0.20+) or BlockPoolSliceScanner (0.23+), is cast to an integer before its sent to a Random.nextInt(...) call. For sufficiently large values of the long 'period', the casted integer may be negative. This is not accounted for. I'll attach a sample test that shows this possibility with the numbers.

      We should ensure we do a Math.abs(...) before we send it to the Random.nextInt(...) call to avoid this.

      With this bug, the maximum # of blocks a scanner may hold in its blocksMap without opening up the chance for beginning this exception (intermittent, as blocks continue to grow) would be 3582718.

      1. BSBugTest.java
        0.5 kB
        Harsh J
      2. HDFS-2541.patch
        1 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Patch and sample test attached.

          Show
          Harsh J added a comment - Patch and sample test attached.
          Hide
          Harsh J added a comment -

          (Also noted that the int cast would reach 0 when the period is 134217728, but that's too large to happen on a single DN? Might we worth adding a check for that as well, though.)

          Show
          Harsh J added a comment - (Also noted that the int cast would reach 0 when the period is 134217728, but that's too large to happen on a single DN? Might we worth adding a check for that as well, though.)
          Hide
          Eli Collins added a comment -

          +1 lgtm

          Show
          Eli Collins added a comment - +1 lgtm
          Hide
          Harsh J added a comment -

          Running build + test.

          Show
          Harsh J added a comment - Running build + test.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502585/BSBugTest.java
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1536//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502585/BSBugTest.java against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1536//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Re-upping patch for proper buildbot submission.

          Show
          Harsh J added a comment - Re-upping patch for proper buildbot submission.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502602/HDFS-2541.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1537//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1537//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502602/HDFS-2541.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1537//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1537//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1361 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1361/)
          HDFS-2541. For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1361 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1361/ ) HDFS-2541 . For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Eli Collins added a comment -

          Test failures are unrelated. I committed this and merged to 23 and 206 since this is low risk and we've seen this bug in the wild. Thanks Harsh!

          Show
          Eli Collins added a comment - Test failures are unrelated. I committed this and merged to 23 and 206 since this is low risk and we've seen this bug in the wild. Thanks Harsh!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #184 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/184/)
          HDFS-2541. svn merge -c 1204114 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #184 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/184/ ) HDFS-2541 . svn merge -c 1204114 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1287 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1287/)
          HDFS-2541. For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1287 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1287/ ) HDFS-2541 . For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #183 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/183/)
          HDFS-2541. svn merge -c 1204114 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #183 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/183/ ) HDFS-2541 . svn merge -c 1204114 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/196/)
          HDFS-2541. svn merge -c 1204114 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/196/ ) HDFS-2541 . svn merge -c 1204114 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1313 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1313/)
          HDFS-2541. For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1313 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1313/ ) HDFS-2541 . For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Harsh J added a comment -

          Thanks Eli!

          Would it also make sense if we 'warn' when block #s surpass a particularly large value? Could open a new ticket for this if you think it makes sense to do that.

          Show
          Harsh J added a comment - Thanks Eli! Would it also make sense if we 'warn' when block #s surpass a particularly large value? Could open a new ticket for this if you think it makes sense to do that.
          Hide
          Eli Collins added a comment -

          Good idea, please open a new ticket.

          Show
          Eli Collins added a comment - Good idea, please open a new ticket.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/)
          HDFS-2541. For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #869 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/869/ ) HDFS-2541 . For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #82 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/82/)
          HDFS-2541. svn merge -c 1204114 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #82 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/82/ ) HDFS-2541 . svn merge -c 1204114 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/99/)
          HDFS-2541. svn merge -c 1204114 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/99/ ) HDFS-2541 . svn merge -c 1204114 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204115 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/)
          HDFS-2541. For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #903 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/903/ ) HDFS-2541 . For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value. Contributed by Harsh J eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204114 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java
          Hide
          Colin Patrick McCabe added a comment -

          Eeek. It seems like setting period = 0xffffffffL will result in a very short period being set, contrary to the intended effect.

          Show
          Colin Patrick McCabe added a comment - Eeek. It seems like setting period = 0xffffffffL will result in a very short period being set, contrary to the intended effect.
          Hide
          Colin Patrick McCabe added a comment -

          Sorry, the example I meant to give is that with period = 0x100000000L, we'll "truncate" it to 0. That can't possibly be right.

          Show
          Colin Patrick McCabe added a comment - Sorry, the example I meant to give is that with period = 0x100000000L, we'll "truncate" it to 0. That can't possibly be right.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Colin, good point. The fix is incorrect. Do you want to create a JIRA?

          Show
          Tsz Wo Nicholas Sze added a comment - Colin, good point. The fix is incorrect. Do you want to create a JIRA?
          Hide
          Colin Patrick McCabe added a comment -

          Filed HDFS-3488

          Show
          Colin Patrick McCabe added a comment - Filed HDFS-3488
          Hide
          Rajat Agarwal added a comment -
          long period = Math.min(scanPeriod, 
                                     Math.max(blockMap.size(),1) * 600 * 1000L);
          

          The problem here is that when the blockMap.size becomes >3579139 then the result of its multiplication with 600 exceeds the INTEGER.MAX_VALUE and gives negative value. Had it been like

          long period = Math.min(scanPeriod, 
                                     Math.max(blockMap.size(),1) * 600L * 1000L);
          

          "period" would have been dependent on scanPeriod only OR always positive.

          Now there can be two cases here:

          1) If scanPeriod is taken up the default value (21*24L three weeks) or its less than 596 hours (596*3600*1000 < INTEGER.MAX_VALUE) the "period" would always been a positive value.
          2) If the scan period is more than that, the "period" would again be negative. In this case we can have something like this

          if((int)period < 0)
              period=scanPeriod
          
          return System.currentTimeMillis() - scanPeriod + 
                     random.nextInt((int)period); 
          
          Show
          Rajat Agarwal added a comment - long period = Math .min(scanPeriod, Math .max(blockMap.size(),1) * 600 * 1000L); The problem here is that when the blockMap.size becomes >3579139 then the result of its multiplication with 600 exceeds the INTEGER.MAX_VALUE and gives negative value. Had it been like long period = Math .min(scanPeriod, Math .max(blockMap.size(),1) * 600L * 1000L); "period" would have been dependent on scanPeriod only OR always positive. Now there can be two cases here: 1) If scanPeriod is taken up the default value (21*24L three weeks) or its less than 596 hours (596*3600*1000 < INTEGER.MAX_VALUE) the "period" would always been a positive value. 2) If the scan period is more than that, the "period" would again be negative. In this case we can have something like this if (( int )period < 0) period=scanPeriod return System .currentTimeMillis() - scanPeriod + random.nextInt(( int )period);
          Hide
          Harsh J added a comment -

          Hi Rajat,

          Thanks! You may want to file a new JIRA with the identified bug if it is not already fixed in trunk and is not covered by Nicholas and Colin's HDFS-3456 (if it is, please post there); since this is a closed issue.

          Show
          Harsh J added a comment - Hi Rajat, Thanks! You may want to file a new JIRA with the identified bug if it is not already fixed in trunk and is not covered by Nicholas and Colin's HDFS-3456 (if it is, please post there); since this is a closed issue.
          Hide
          Colin Patrick McCabe added a comment -

          Check out the patch I posted to HDFS-3488.

          Show
          Colin Patrick McCabe added a comment - Check out the patch I posted to HDFS-3488 .

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development