Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3091

Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters.

    Details

      Description

      When verifying the HDFS-1606 feature, Observed couple of issues.

      Presently the ReplaceDatanodeOnFailure policy satisfies even though we dont have enough DN to replcae in cluster and will be resulted into write failure.

      12/03/13 14:27:12 WARN hdfs.DFSClient: DataStreamer Exception
      java.io.IOException: Failed to add a datanode: nodes.length != original.length + 1, nodes=[xx.xx.xx.xx:50010], original=[xx.xx.xx.xx1:50010]
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:778)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:834)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:930)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:741)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:416)

      Lets take some cases:
      1) Replication factor 3 and cluster size also 3 and unportunately pipeline drops to 1.

      ReplaceDatanodeOnFailure will be satisfied because existings(1)<= replication/2 (3/2==1).

      But when it finding the new node to replace obiously it can not find the new node and the sanity check will fail.

      This will be resulted to Wite failure.

      2) Replication factor 10 (accidentally user sets the replication factor to higher value than cluster size),
      Cluser has only 5 datanodes.

      Here even if one node fails also write will fail with same reason.
      Because pipeline max will be 5 and killed one datanode, then existings will be 4

      existings(4)<= replication/2(10/2==5) will be satisfied and obiously it can not replace with the new node as there is no extra nodes exist in the cluster. This will be resulted to write failure.

      3) sync realted opreations also fails in this situations ( will post the clear scenarios)

      1. h3091_20120319.patch
        0.8 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          @Zhanwei, let me reply you on HDFS-3179.

          Show
          Tsz Wo Nicholas Sze added a comment - @Zhanwei, let me reply you on HDFS-3179 .
          Hide
          Zhanwei.Wang added a comment -

          Hi, Nocholas

          I would say the failures are expected. The feature is to guarantee the number of replicas that the user is asking. However, the cluster is too small that the guarantee is impossible. It makes sense to fail the write requests.

          I agree with you, but have a look at code. in HDFS-3179, I first create a EMPTY file and append twice, the first append finished successfully but the second failed since there is only one datanode and the number of replica is 3.

          Is that what you want to see? I think the policy check should fail on the first write to the file.

          Show
          Zhanwei.Wang added a comment - Hi, Nocholas I would say the failures are expected. The feature is to guarantee the number of replicas that the user is asking. However, the cluster is too small that the guarantee is impossible. It makes sense to fail the write requests. I agree with you, but have a look at code. in HDFS-3179 , I first create a EMPTY file and append twice, the first append finished successfully but the second failed since there is only one datanode and the number of replica is 3. Is that what you want to see? I think the policy check should fail on the first write to the file.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/)
          HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/ ) HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/)
          Merge HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633)

          Result = FAILURE
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          • /hadoop/common/branches/branch-0.23/hadoop-project
          • /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/ ) Merge HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job /hadoop/common/branches/branch-0.23/hadoop-project /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #710 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/710/)
          Merge HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633)

          Result = ABORTED
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          • /hadoop/common/branches/branch-0.23/hadoop-project
          • /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #710 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/710/ ) Merge HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633) Result = ABORTED umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job /hadoop/common/branches/branch-0.23/hadoop-project /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1908 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1908/)
          HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624)

          Result = ABORTED
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1908 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1908/ ) HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624) Result = ABORTED umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #702 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/702/)
          Merge HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          • /hadoop/common/branches/branch-0.23/hadoop-project
          • /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #702 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/702/ ) Merge HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job /hadoop/common/branches/branch-0.23/hadoop-project /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #693 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/693/)
          Merge HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633
          Files :

          • /hadoop/common/branches/branch-0.23
          • /hadoop/common/branches/branch-0.23/hadoop-common-project
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job
          • /hadoop/common/branches/branch-0.23/hadoop-project
          • /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #693 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/693/ ) Merge HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302633) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302633 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job /hadoop/common/branches/branch-0.23/hadoop-project /hadoop/common/branches/branch-0.23/hadoop-project/src/site
          Hide
          Uma Maheswara Rao G added a comment -

          Just merged to 0.23 also.

          Show
          Uma Maheswara Rao G added a comment - Just merged to 0.23 also.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1974 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1974/)
          HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1974 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1974/ ) HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1900 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1900/)
          HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1900 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1900/ ) HDFS-3091 . Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters. Contributed by Nicholas. (Revision 1302624) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302624 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Uma Maheswara Rao G added a comment -

          tomorrow, I will back-port this to 0.23 branch as well.

          Show
          Uma Maheswara Rao G added a comment - tomorrow, I will back-port this to 0.23 branch as well.
          Hide
          Uma Maheswara Rao G added a comment -

          Updated the title.

          Committed to trunk. Thanks Nicholas for the patch.

          Show
          Uma Maheswara Rao G added a comment - Updated the title. Committed to trunk. Thanks Nicholas for the patch.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot Nicholas.
          Patch looks good.
          +1

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot Nicholas. Patch looks good. +1
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h3091_20120319.patch: add comments for small clusters.

          Show
          Tsz Wo Nicholas Sze added a comment - h3091_20120319.patch: add comments for small clusters.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > What if we log about the cluster size and can recommend to disable this feature with smaller clusters? ...

          Sure, let's add some comments for this.

          > assume cluster size is 10,user set the replication values to 10, when it have been writen 9,one of them is bad, do you think that its writing is success?

          If the DEFAULT policy is used, the pipeline won't fail until the number of datanodes N drops to 5 as described in (2) in the description. In your example, if the user set replication to 18 and N drops to 9, the write should fail.

          Show
          Tsz Wo Nicholas Sze added a comment - > What if we log about the cluster size and can recommend to disable this feature with smaller clusters? ... Sure, let's add some comments for this. > assume cluster size is 10,user set the replication values to 10, when it have been writen 9,one of them is bad, do you think that its writing is success? If the DEFAULT policy is used, the pipeline won't fail until the number of datanodes N drops to 5 as described in (2) in the description. In your example, if the user set replication to 18 and N drops to 9, the write should fail.
          Hide
          liaowenrui added a comment -

          yeah,I agree with him! thank you for your answer!

          its designing idea is good,But this implement have a defect.

          I think this featrue is to guarantee replication in reliability.

          assume cluster size is 10,user set the replication values to 10, when it have been writen 9,one of them is bad, do you think that its writing is success?

          Show
          liaowenrui added a comment - yeah,I agree with him! thank you for your answer! its designing idea is good,But this implement have a defect. I think this featrue is to guarantee replication in reliability. assume cluster size is 10,user set the replication values to 10, when it have been writen 9,one of them is bad, do you think that its writing is success?
          Hide
          Uma Maheswara Rao G added a comment -

          Does it make sense to you?

          Yes, it make sense to me. There is no point of enabling this feature for smaller clusters. This feature will give real advantages only when we go into big clusters.

          What if we log about the cluster size and can recommend to disable this feature with smaller clusters? because by default this feature is enabled, when user testing with 2-3 nodes clusters, this may fail due to pipeline failure and will be resulted to write failure.

          Failed to add a datanode: nodes.length != original.length + 1, nodes=[10.18.52.55:50010], original=[10.18.52.55:50010]

          for example:
          Cluster size and replication factor nearly equal, there is no extra node available in the cluster to add the new node into this pipeline. Hadoop recommends to set dfs.client.block.write.replace-datanode-on-failure.policy to NEVER for smaller clusters.

          @liaowenrui, do you agree with Nicholas?

          Show
          Uma Maheswara Rao G added a comment - Does it make sense to you? Yes, it make sense to me. There is no point of enabling this feature for smaller clusters. This feature will give real advantages only when we go into big clusters. What if we log about the cluster size and can recommend to disable this feature with smaller clusters? because by default this feature is enabled, when user testing with 2-3 nodes clusters, this may fail due to pipeline failure and will be resulted to write failure. Failed to add a datanode: nodes.length != original.length + 1, nodes= [10.18.52.55:50010] , original= [10.18.52.55:50010] for example: Cluster size and replication factor nearly equal, there is no extra node available in the cluster to add the new node into this pipeline. Hadoop recommends to set dfs.client.block.write.replace-datanode-on-failure.policy to NEVER for smaller clusters. @liaowenrui, do you agree with Nicholas?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I think when the block's replication is greater or equal than min replication number and not any datanode can be choosen,the block is written success.

          One of the reasons having this feature is to avoid writing to a single replica but the statement above allow it.

          Show
          Tsz Wo Nicholas Sze added a comment - > I think when the block's replication is greater or equal than min replication number and not any datanode can be choosen,the block is written success. One of the reasons having this feature is to avoid writing to a single replica but the statement above allow it.
          Hide
          liaowenrui added a comment -

          hi tsz,

          you say it is not reasonable.

          this issue is doesn't matter with the small or the big cluster.

          I think when the block's replication is greater or equal than min replication number and not any datanode can be choosen,the block is written success.

          Show
          liaowenrui added a comment - hi tsz, you say it is not reasonable. this issue is doesn't matter with the small or the big cluster. I think when the block's replication is greater or equal than min replication number and not any datanode can be choosen,the block is written success.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Uma,

          First of all, thanks for testing it.

          I would say the failures are expected. The feature is to guarantee the number of replicas that the user is asking. However, the cluster is too small that the guarantee is impossible. It makes sense to fail the write requests.

          Note that policy is a client side configuration. The user could set the policy to NEVER. For the 3-node case, the admin should disable the feature (or set the policy to NEVER in the default conf.)

          Does it make sense to you?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Uma, First of all, thanks for testing it. I would say the failures are expected. The feature is to guarantee the number of replicas that the user is asking. However, the cluster is too small that the guarantee is impossible. It makes sense to fail the write requests. Note that policy is a client side configuration. The user could set the policy to NEVER. For the 3-node case, the admin should disable the feature (or set the policy to NEVER in the default conf.) Does it make sense to you?
          Hide
          Uma Maheswara Rao G added a comment -

          I am thinking about this issue,
          I think if the replication is greater or equal to the total live nodes in the cluser, then we need not go for replacing the pipeline with new node.

          But here we may need to do RPC for checking the live nodes in the cluster.

          Any alternative solutions?

          Also other case is, if NN not able to choose any extra nodes due to DN exeiver count or other factors, then also it may fail right? In this case, i feel that sanity check itself may not be correct check right. Can't we simply proceed as normal behaviour when we are not able to find any new node from NN? Why do we need to strctily ensure one extra node and make write failure?

          @Nocholas, Since you are the author for this feature, I need your opinion.

          Show
          Uma Maheswara Rao G added a comment - I am thinking about this issue, I think if the replication is greater or equal to the total live nodes in the cluser, then we need not go for replacing the pipeline with new node. But here we may need to do RPC for checking the live nodes in the cluster. Any alternative solutions? Also other case is, if NN not able to choose any extra nodes due to DN exeiver count or other factors, then also it may fail right? In this case, i feel that sanity check itself may not be correct check right. Can't we simply proceed as normal behaviour when we are not able to find any new node from NN? Why do we need to strctily ensure one extra node and make write failure? @Nocholas, Since you are the author for this feature, I need your opinion.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development