Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2981

The default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There was a typo earlier in the default value of dfs.client.block.write.replace-datanode-on-failure.enable. Then, HDFS-2944 changed from "ture" to "false". It should be changed to "true".

      1. h2981_20120221.patch
        0.6 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2981_20120221.patch: changed the default to true.

          Show
          Tsz Wo Nicholas Sze added a comment - h2981_20120221.patch: changed the default to true.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12515484/h2981_20120221.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1888//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1888//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515484/h2981_20120221.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1888//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1888//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          I disagree that it should be true by default. Apps like HBase which are latency-sensitive don't want to wait for a whole block to be re-transferred when a node in the pipeline fails. Apps which are long-running writes and non-latency-sensitive (eg log collection) can flip this to true on their own client, no?

          Show
          Todd Lipcon added a comment - I disagree that it should be true by default. Apps like HBase which are latency-sensitive don't want to wait for a whole block to be re-transferred when a node in the pipeline fails. Apps which are long-running writes and non-latency-sensitive (eg log collection) can flip this to true on their own client, no?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd,

          Enabling the feature does not mean that re-transferring a block when a node in the pipeline fail. There is another conf property, dfs.client.block.write.replace-datanode-on-failure.policy, for configuring the policy. The default is

              DEFAULT: 
                Let r be the replication number.
                Let n be the number of existing datanodes.
                Add a new datanode only if r is greater than or equal to 3 and either
                (1) floor(r/2) is greater than or equal to n; or
                (2) r is greater than n and the block is hflushed/appended.
          

          Also, individual applications can set the policy to NEVER if it is desirable.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, Enabling the feature does not mean that re-transferring a block when a node in the pipeline fail. There is another conf property, dfs.client.block.write.replace-datanode-on-failure.policy, for configuring the policy. The default is DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended. Also, individual applications can set the policy to NEVER if it is desirable.
          Hide
          Todd Lipcon added a comment -

          Oh, I see. That makes sense, then...

          But I think you need to change the default in the code as well, right?

          Show
          Todd Lipcon added a comment - Oh, I see. That makes sense, then... But I think you need to change the default in the code as well, right?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The default in code currently is set to true.

          //DFSConfigKeys
            public static final boolean DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_ENABLE_DEFAULT = true;
          
          Show
          Tsz Wo Nicholas Sze added a comment - The default in code currently is set to true. //DFSConfigKeys public static final boolean DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_ENABLE_DEFAULT = true ;
          Hide
          Todd Lipcon added a comment -

          Oh, my bad. Sorry for not looking. +1

          Show
          Todd Lipcon added a comment - Oh, my bad. Sorry for not looking. +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12515484/h2981_20120221.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1891//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1891//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515484/h2981_20120221.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1891//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1891//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Todd, thanks for taking a look.

          I have committed this to trunk and 0.23.

          Show
          Tsz Wo Nicholas Sze added a comment - Todd, thanks for taking a look. I have committed this to trunk and 0.23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #585 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/585/)
          svn merge -c 1292626 from trunk for HDFS-2981. (Revision 1292627)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #585 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/585/ ) svn merge -c 1292626 from trunk for HDFS-2981 . (Revision 1292627) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1765 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1765/)
          HDFS-2981. In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1765 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1765/ ) HDFS-2981 . In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #572 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/572/)
          svn merge -c 1292626 from trunk for HDFS-2981. (Revision 1292627)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #572 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/572/ ) svn merge -c 1292626 from trunk for HDFS-2981 . (Revision 1292627) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1839 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1839/)
          HDFS-2981. In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1839 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1839/ ) HDFS-2981 . In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1776 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1776/)
          HDFS-2981. In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626)

          Result = ABORTED
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1776 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1776/ ) HDFS-2981 . In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #587 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/587/)
          svn merge -c 1292626 from trunk for HDFS-2981. (Revision 1292627)

          Result = ABORTED
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #587 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/587/ ) svn merge -c 1292626 from trunk for HDFS-2981 . (Revision 1292627) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #964 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/964/)
          HDFS-2981. In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #964 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/964/ ) HDFS-2981 . In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #177 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/177/)
          svn merge -c 1292626 from trunk for HDFS-2981. (Revision 1292627)

          Result = UNSTABLE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #177 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/177/ ) svn merge -c 1292626 from trunk for HDFS-2981 . (Revision 1292627) Result = UNSTABLE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #205 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/205/)
          svn merge -c 1292626 from trunk for HDFS-2981. (Revision 1292627)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #205 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/205/ ) svn merge -c 1292626 from trunk for HDFS-2981 . (Revision 1292627) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #999 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/999/)
          HDFS-2981. In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #999 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/999/ ) HDFS-2981 . In hdfs-default.xml, the default value of dfs.client.block.write.replace-datanode-on-failure.enable should be true. (Revision 1292626) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development