Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11799

Introduce a config to allow setting up write pipeline with fewer nodes than replication factor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added new configuration "dfs.client.block.write.replace-datanode-on-failure.min-replication".
           
          The minimum number of replications that are needed to not to fail
            the write pipeline if new datanodes can not be found to replace
            failed datanodes (could be due to network failure) in the write pipeline.
            If the number of the remaining datanodes in the write pipeline is greater
            than or equal to this property value, continue writing to the remaining nodes.
            Otherwise throw exception.

            If this is set to 0, an exception will be thrown, when a replacement
            can not be found.
          
      Show
      Added new configuration "dfs.client.block.write.replace-datanode-on-failure.min-replication".           The minimum number of replications that are needed to not to fail       the write pipeline if new datanodes can not be found to replace       failed datanodes (could be due to network failure) in the write pipeline.       If the number of the remaining datanodes in the write pipeline is greater       than or equal to this property value, continue writing to the remaining nodes.       Otherwise throw exception.       If this is set to 0, an exception will be thrown, when a replacement       can not be found.     

      Description

      During pipeline recovery, if not enough DNs can be found, if
      dfs.client.block.write.replace-datanode-on-failure.best-effort
      is enabled, we let the pipeline to continue, even if there is a single DN.

      Similarly, when we create the write pipeline initially, if for some reason we can't find enough DNs, we can have a similar config to enable writing with a single DN.

      More study will be done.

      1. HDFS-11799.patch
        21 kB
        Brahma Reddy Battula
      2. HDFS-11799-002.patch
        22 kB
        Brahma Reddy Battula
      3. HDFS-11799-003.patch
        22 kB
        Brahma Reddy Battula
      4. HDFS-11799-004.patch
        17 kB
        Brahma Reddy Battula
      5. HDFS-11799-005.patch
        17 kB
        Brahma Reddy Battula
      6. HDFS-11799-006.patch
        17 kB
        Brahma Reddy Battula
      7. HDFS-11799-007.patch
        17 kB
        Brahma Reddy Battula
      8. HDFS-11799-008.patch
        17 kB
        Brahma Reddy Battula
      9. HDFS-11799-009.patch
        17 kB
        Brahma Reddy Battula

        Activity

        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Yongjun Zhang Nice Idea..
        Configuration can be dfs.client.block.write.replace-datanode-on-failure.replication ..?
        We had similar approach which was used for small cluster..will update patch tomorrow.

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Yongjun Zhang Nice Idea.. Configuration can be dfs.client.block.write.replace-datanode-on-failure.replication ..? We had similar approach which was used for small cluster..will update patch tomorrow.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Initial draft..Kindly review.

        Pipeline will continue based on the newly added config.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Initial draft..Kindly review. Pipeline will continue based on the newly added config.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Yongjun Zhang can you please review patch ..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang can you please review patch ..?
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Brahma Reddy Battula,

        Thanks for working on this and sorry for the delayed review.

        Some comments:

        1. Suggest to add some comments about dtpReplaceDatanodeOnFailureReplication when it was introduced. And add a default config with description in hdfs-default.xml
        2. about the following code

             final int d;
              try {
                d = findNewDatanode(original);
              } catch (IOException ioe) {
                if (dfsClient.dtpReplaceDatanodeOnFailureReplication > 0 && nodes.length
                    >= dfsClient.dtpReplaceDatanodeOnFailureReplication) {
                  DFSClient.LOG.warn(
                      "Failed to add a new datanode for write pipeline, minimum block replication:"
                          + dfsClient.dtpReplaceDatanodeOnFailureReplication
                          + ", good datanode size: " + nodes.length);
                  return;
                }
                throw ioe;
              }
        

        2.1 add comment explaining things
        2.2 add the ioe to the warn, so to expose the ioe in the log

        3. Any chance to have a unit test?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Brahma Reddy Battula , Thanks for working on this and sorry for the delayed review. Some comments: 1. Suggest to add some comments about dtpReplaceDatanodeOnFailureReplication when it was introduced. And add a default config with description in hdfs-default.xml 2. about the following code final int d; try { d = findNewDatanode(original); } catch (IOException ioe) { if (dfsClient.dtpReplaceDatanodeOnFailureReplication > 0 && nodes.length >= dfsClient.dtpReplaceDatanodeOnFailureReplication) { DFSClient.LOG.warn( "Failed to add a new datanode for write pipeline, minimum block replication:" + dfsClient.dtpReplaceDatanodeOnFailureReplication + ", good datanode size: " + nodes.length); return ; } throw ioe; } 2.1 add comment explaining things 2.2 add the ioe to the warn, so to expose the ioe in the log 3. Any chance to have a unit test? Thanks.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Yongjun Zhang thanks a lot for taking look..

        Uploaded the patch to address your comments apart from the following.

        3. Any chance to have a unit test?

        Testclass already present TestReplaceDatanodeFailureReplication.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang thanks a lot for taking look.. Uploaded the patch to address your comments apart from the following. 3. Any chance to have a unit test? Testclass already present TestReplaceDatanodeFailureReplication .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 15m 53s trunk passed
        +1 compile 1m 47s trunk passed
        +1 checkstyle 0m 51s trunk passed
        +1 mvnsite 1m 44s trunk passed
        +1 findbugs 3m 35s trunk passed
        +1 javadoc 1m 10s trunk passed
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 39s the patch passed
        +1 compile 1m 47s the patch passed
        +1 javac 1m 47s the patch passed
        -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 18 new + 556 unchanged - 0 fixed = 574 total (was 556)
        +1 mvnsite 1m 41s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 47s the patch passed
        +1 javadoc 1m 6s the patch passed
        +1 unit 1m 23s hadoop-hdfs-client in the patch passed.
        -1 unit 83m 2s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        122m 55s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.web.TestWebHdfsTimeouts



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874360/HDFS-11799-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 30e2f2301a4b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0111711
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20030/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20030/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20030/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20030/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 15m 53s trunk passed +1 compile 1m 47s trunk passed +1 checkstyle 0m 51s trunk passed +1 mvnsite 1m 44s trunk passed +1 findbugs 3m 35s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 1m 47s the patch passed +1 javac 1m 47s the patch passed -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 18 new + 556 unchanged - 0 fixed = 574 total (was 556) +1 mvnsite 1m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 1m 23s hadoop-hdfs-client in the patch passed. -1 unit 83m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 122m 55s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.web.TestWebHdfsTimeouts Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874360/HDFS-11799-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 30e2f2301a4b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20030/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20030/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20030/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20030/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Updated the patch for fixable checksytel issues.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Updated the patch for fixable checksytel issues.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 31s Maven dependency ordering for branch
        +1 mvninstall 14m 40s trunk passed
        +1 compile 1m 29s trunk passed
        +1 checkstyle 0m 48s trunk passed
        +1 mvnsite 1m 29s trunk passed
        +1 findbugs 3m 22s trunk passed
        +1 javadoc 1m 2s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 21s the patch passed
        +1 compile 1m 25s the patch passed
        +1 javac 1m 25s the patch passed
        -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 6 new + 556 unchanged - 0 fixed = 562 total (was 556)
        +1 mvnsite 1m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 35s the patch passed
        +1 javadoc 0m 57s the patch passed
        +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
        -1 unit 66m 51s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        102m 56s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874365/HDFS-11799-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 714b1af4d156 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0111711
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20031/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20031/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 31s Maven dependency ordering for branch +1 mvninstall 14m 40s trunk passed +1 compile 1m 29s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 29s trunk passed +1 findbugs 3m 22s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 25s the patch passed +1 javac 1m 25s the patch passed -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 6 new + 556 unchanged - 0 fixed = 562 total (was 556) +1 mvnsite 1m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 35s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 66m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 102m 56s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874365/HDFS-11799-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 714b1af4d156 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20031/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20031/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        HI Brahma Reddy Battula,

        Thanks for the updated patch and really worry for the delayed review. It in general looks pretty good. I have some comments about cosmetics things:

        1. Suggest to change

        DFSClient.LOG.warn(
                      "Failed to add a new datanode for write pipeline, minimum block "
                          + "replication:"
                          + dfsClient.dtpReplaceDatanodeOnFailureReplication
                          + ", good datanode size: " + nodes.length, ioe);
        

        to

        DFSClient.LOG.warn(
                      "Failed to find a new datanode to add to the write pipeline, "
                          " continue to write to the pipeline with " + nodes.length
                          " nodes since it's bigger than minimum replication: "
                          + dfsClient.dtpReplaceDatanodeOnFailureReplication
                          + " configued by "
                          + HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION,
                          + ".", ioe);
        

        2. Seems we don't need to add two new entries and call them deprecated:

          @Deprecated
          public static final String DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION =
              HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION;
          @Deprecated
          public static final short DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION_DEFAULT =
              HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION_DEFAULT;
        

        So we can drop them right?

        3. in hdfs-default.xml, what about change it to:

              The minimum number of replications that are needed to not to fail
              the write pipeline if new datanodes can not be found to replace
              failed datanodes (could be due to network failure) in the write pipeline.
              If the number of the remaining datanodes in the write pipeline is greater than or
              equal to this property value, continue writing to the remaining nodes.
              Otherwise throw exception.
        
              If this is set to 0 or a negative number, an exception will be thrown
              when a replacement can not be found.
              See also dfs.client.block.write.replace-datanode-on-failure.policy
        

        4. Thanks for the comprehensive tests. I notice that the comment of test code sometimes don't match the code, e.g.

        54   /** Test fail all the datanodes except first in the pipeline. */
        255   @Test public void testWithOnlyLastDatanodeIsAlive() throws Exception {
        

        and

        317   /** Test fail all the datanodes except first in the pipeline. */
        318   @Test
        319   public void testLessNumberOfLiveDatanodesThanWriteReplaceDatanodeOnFailureRF()
        

        5. Possible to consolidate simliar part of the tests into one helper method with a parameter to indicate which
        DNs to stop, to avoid redundant code?

        6. Seems even for the test that exception is thrown due to failure to find new datanode, the
        result file can still be verified to have the expected full content with verifyFileContent()?

        Show
        yzhangal Yongjun Zhang added a comment - HI Brahma Reddy Battula , Thanks for the updated patch and really worry for the delayed review. It in general looks pretty good. I have some comments about cosmetics things: 1. Suggest to change DFSClient.LOG.warn( "Failed to add a new datanode for write pipeline, minimum block " + "replication:" + dfsClient.dtpReplaceDatanodeOnFailureReplication + ", good datanode size: " + nodes.length, ioe); to DFSClient.LOG.warn( "Failed to find a new datanode to add to the write pipeline, " " continue to write to the pipeline with " + nodes.length " nodes since it's bigger than minimum replication: " + dfsClient.dtpReplaceDatanodeOnFailureReplication + " configued by " + HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION, + "." , ioe); 2. Seems we don't need to add two new entries and call them deprecated: @Deprecated public static final String DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION = HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION; @Deprecated public static final short DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION_DEFAULT = HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION_DEFAULT; So we can drop them right? 3. in hdfs-default.xml, what about change it to: The minimum number of replications that are needed to not to fail the write pipeline if new datanodes can not be found to replace failed datanodes (could be due to network failure) in the write pipeline. If the number of the remaining datanodes in the write pipeline is greater than or equal to this property value, continue writing to the remaining nodes. Otherwise throw exception. If this is set to 0 or a negative number, an exception will be thrown when a replacement can not be found. See also dfs.client.block.write.replace-datanode-on-failure.policy 4. Thanks for the comprehensive tests. I notice that the comment of test code sometimes don't match the code, e.g. 54 /** Test fail all the datanodes except first in the pipeline. */ 255 @Test public void testWithOnlyLastDatanodeIsAlive() throws Exception { and 317 /** Test fail all the datanodes except first in the pipeline. */ 318 @Test 319 public void testLessNumberOfLiveDatanodesThanWriteReplaceDatanodeOnFailureRF() 5. Possible to consolidate simliar part of the tests into one helper method with a parameter to indicate which DNs to stop, to avoid redundant code? 6. Seems even for the test that exception is thrown due to failure to find new datanode, the result file can still be verified to have the expected full content with verifyFileContent()?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded patch to address the comments

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded patch to address the comments
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 12m 36s trunk passed
        +1 compile 1m 17s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 22s trunk passed
        -1 findbugs 1m 19s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
        -1 findbugs 1m 33s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 55s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 11s the patch passed
        +1 compile 1m 14s the patch passed
        -1 javac 1m 14s hadoop-hdfs-project generated 1 new + 430 unchanged - 0 fixed = 431 total (was 430)
        -0 checkstyle 0m 43s hadoop-hdfs-project: The patch generated 6 new + 556 unchanged - 0 fixed = 562 total (was 556)
        +1 mvnsite 1m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 58s the patch passed
        +1 javadoc 0m 54s the patch passed
              Other Tests
        +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
        -1 unit 65m 27s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        97m 20s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.tools.TestHdfsConfigFields
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878805/HDFS-11799-004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 08d364792b38 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac9489f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20408/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20408/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 12m 36s trunk passed +1 compile 1m 17s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 22s trunk passed -1 findbugs 1m 19s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 33s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 55s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 1m 14s the patch passed -1 javac 1m 14s hadoop-hdfs-project generated 1 new + 430 unchanged - 0 fixed = 431 total (was 430) -0 checkstyle 0m 43s hadoop-hdfs-project: The patch generated 6 new + 556 unchanged - 0 fixed = 562 total (was 556) +1 mvnsite 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 58s the patch passed +1 javadoc 0m 54s the patch passed       Other Tests +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 65m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 97m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878805/HDFS-11799-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 08d364792b38 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac9489f Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20408/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20408/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20408/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        will address Jenkins failures once after review. Yongjun Zhang can you look at latest patch.

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited will address Jenkins failures once after review. Yongjun Zhang can you look at latest patch.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Sorry for the delay Brahma Reddy Battula, here are my comments, largely cosmetic:

        1. HdfsClientConfigKeys
        1.1. String REPLICATION = PREFIX + "replication";
        Need a better name than "replication", maybe "min-setup-replication"?
        1.2 Change hdfs-default accordingly

        2.DataStreamer

        2.1 change

               //check the minimal numbers nodes available to continue the wrtie.
        

        to

               // check the minimal number of nodes available to decide whether to
               // continue the wrtie.
        

        2.2 change

                // threshold value, if yes continue writing to the two remaining nodes.
        

        to

                // threshold value, continue writing to the remaining nodes.
        

        2.3 Change

                          + " nodes since it's bigger than minimum replication: "
                          + dfsClient.dtpReplaceDatanodeOnFailureReplication
                          + " configued by "
        

        to

                          + " nodes since it's no less than minimum setup replication: "
                          + dfsClient.dtpReplaceDatanodeOnFailureReplication
                          + " configured by "
        

        3. hdfs-default.xml

        Suggest to change
        "If this is set to 0 or a negative number, an exception will be thrown
        when a replacement can not be found."
        to
        "If this is set to 0, an exception will be thrown, when a replacement
        can not be found."
        and the code accordingly. Not supporting negative number here seems less
        confusing.

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Sorry for the delay Brahma Reddy Battula , here are my comments, largely cosmetic: 1. HdfsClientConfigKeys 1.1. String REPLICATION = PREFIX + "replication"; Need a better name than "replication", maybe "min-setup-replication"? 1.2 Change hdfs-default accordingly 2.DataStreamer 2.1 change //check the minimal numbers nodes available to continue the wrtie. to // check the minimal number of nodes available to decide whether to // continue the wrtie. 2.2 change // threshold value, if yes continue writing to the two remaining nodes. to // threshold value, continue writing to the remaining nodes. 2.3 Change + " nodes since it's bigger than minimum replication: " + dfsClient.dtpReplaceDatanodeOnFailureReplication + " configued by " to + " nodes since it's no less than minimum setup replication: " + dfsClient.dtpReplaceDatanodeOnFailureReplication + " configured by " 3. hdfs-default.xml Suggest to change "If this is set to 0 or a negative number, an exception will be thrown when a replacement can not be found." to "If this is set to 0, an exception will be thrown, when a replacement can not be found." and the code accordingly. Not supporting negative number here seems less confusing. Thanks.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch to address above comments. now named like "dfs.client.block.write.replace-datanode-on-failure.min.replication" to unify with other.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to address above comments. now named like "dfs.client.block.write.replace-datanode-on-failure.min.replication" to unify with other.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 1s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 16m 11s trunk passed
        +1 compile 1m 40s trunk passed
        +1 checkstyle 0m 49s trunk passed
        +1 mvnsite 1m 40s trunk passed
        +1 findbugs 3m 23s trunk passed
        +1 javadoc 1m 8s trunk passed
              Patch Compile Tests
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 34s the patch passed
        +1 compile 1m 37s the patch passed
        +1 javac 1m 37s the patch passed
        -0 checkstyle 0m 49s hadoop-hdfs-project: The patch generated 4 new + 568 unchanged - 0 fixed = 572 total (was 568)
        +1 mvnsite 1m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 40s the patch passed
        +1 javadoc 1m 4s the patch passed
              Other Tests
        +1 unit 1m 15s hadoop-hdfs-client in the patch passed.
        -1 unit 96m 2s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        134m 58s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.tools.TestHdfsConfigFields
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884458/HDFS-11799-005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 320cd863b68c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 200b113
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20924/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20924/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 16m 11s trunk passed +1 compile 1m 40s trunk passed +1 checkstyle 0m 49s trunk passed +1 mvnsite 1m 40s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 8s trunk passed       Patch Compile Tests 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 1m 37s the patch passed +1 javac 1m 37s the patch passed -0 checkstyle 0m 49s hadoop-hdfs-project: The patch generated 4 new + 568 unchanged - 0 fixed = 572 total (was 568) +1 mvnsite 1m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 40s the patch passed +1 javadoc 1m 4s the patch passed       Other Tests +1 unit 1m 15s hadoop-hdfs-client in the patch passed. -1 unit 96m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 134m 58s Reason Tests Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884458/HDFS-11799-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 320cd863b68c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 200b113 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20924/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20924/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch to address checkstyle and testcase fix.

        hadoop.hdfs.TestClientProtocolForPipelineRecovery

        unrelated,there is tracking jira also HDFS-12378

        hadoop.tools.TestHdfsConfigFields

        handled
        Rest all are passing locally.

        I feel, 3 checkstyles can be ignored (2 are more than 80 and VisibilityModifier)

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to address checkstyle and testcase fix. hadoop.hdfs.TestClientProtocolForPipelineRecovery unrelated,there is tracking jira also HDFS-12378 hadoop.tools.TestHdfsConfigFields handled Rest all are passing locally. I feel, 3 checkstyles can be ignored (2 are more than 80 and VisibilityModifier)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 14m 59s trunk passed
        +1 compile 1m 39s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 findbugs 3m 32s trunk passed
        +1 javadoc 1m 10s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 37s the patch passed
        +1 compile 1m 48s the patch passed
        +1 javac 1m 48s the patch passed
        -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 4 new + 158 unchanged - 0 fixed = 162 total (was 158)
        +1 mvnsite 1m 32s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 33s the patch passed
        +1 javadoc 1m 1s the patch passed
              Other Tests
        +1 unit 1m 11s hadoop-hdfs-client in the patch passed.
        -1 unit 94m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        132m 29s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDatanodeConfig
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.cli.TestCryptoAdminCLI
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884662/HDFS-11799-006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux d5555fc8ef90 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac12e15
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20939/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20939/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20939/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20939/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 14m 59s trunk passed +1 compile 1m 39s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 32s trunk passed +1 findbugs 3m 32s trunk passed +1 javadoc 1m 10s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 37s the patch passed +1 compile 1m 48s the patch passed +1 javac 1m 48s the patch passed -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 4 new + 158 unchanged - 0 fixed = 162 total (was 158) +1 mvnsite 1m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 33s the patch passed +1 javadoc 1m 1s the patch passed       Other Tests +1 unit 1m 11s hadoop-hdfs-client in the patch passed. -1 unit 94m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 132m 29s Reason Tests Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDatanodeConfig   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.cli.TestCryptoAdminCLI Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884662/HDFS-11799-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d5555fc8ef90 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac12e15 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20939/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20939/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20939/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20939/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Yongjun Zhang if you get chance ,can you review once?

        I feel,we can push to branch-2.8.2 also since this will be useful

        Show
        brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang if you get chance ,can you review once? I feel,we can push to branch-2.8.2 also since this will be useful
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks for the revised patch Brahma Reddy Battula. Patch 006 looks good except for two nits:

        1. suggest to change HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION to HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.MIN_REPLICATION

        2. replace the string 'dfs.client.block.write.replace-datanode-on-failure.min.replication'
        in the comment with 'HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.MIN_REPLICATION'

        Show
        yzhangal Yongjun Zhang added a comment - Thanks for the revised patch Brahma Reddy Battula . Patch 006 looks good except for two nits: 1. suggest to change HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION to HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.MIN_REPLICATION 2. replace the string 'dfs.client.block.write.replace-datanode-on-failure.min.replication' in the comment with 'HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.MIN_REPLICATION'
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded patch as per above suggestion.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded patch as per above suggestion.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 29s Maven dependency ordering for branch
        +1 mvninstall 15m 31s trunk passed
        +1 compile 1m 42s trunk passed
        +1 checkstyle 0m 45s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 findbugs 3m 32s trunk passed
        +1 javadoc 1m 10s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 42s the patch passed
        +1 compile 1m 51s the patch passed
        +1 javac 1m 51s the patch passed
        -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 7 new + 156 unchanged - 0 fixed = 163 total (was 156)
        +1 mvnsite 1m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 56s the patch passed
        +1 javadoc 1m 6s the patch passed
              Other Tests
        +1 unit 1m 24s hadoop-hdfs-client in the patch passed.
        -1 unit 96m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        135m 47s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.tools.TestDFSZKFailoverController
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887563/HDFS-11799-007.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux a6282f784292 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8d7cc22
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21189/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21189/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21189/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 29s Maven dependency ordering for branch +1 mvninstall 15m 31s trunk passed +1 compile 1m 42s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 35s trunk passed +1 findbugs 3m 32s trunk passed +1 javadoc 1m 10s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 1m 51s the patch passed +1 javac 1m 51s the patch passed -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 7 new + 156 unchanged - 0 fixed = 163 total (was 156) +1 mvnsite 1m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 6s the patch passed       Other Tests +1 unit 1m 24s hadoop-hdfs-client in the patch passed. -1 unit 96m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 135m 47s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.namenode.TestNamenodeRetryCache Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887563/HDFS-11799-007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a6282f784292 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d7cc22 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21189/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21189/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21189/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Testfailures are unrelated. Kindly Review.

        TestNamenodeRetryCache and TestRetryCacheWithHA failed after HDFS-12395
        TestLeaseRecoveryStriped is tracked HDFS-12437.

        Rest are passing locally.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Testfailures are unrelated. Kindly Review. TestNamenodeRetryCache and TestRetryCacheWithHA failed after HDFS-12395 TestLeaseRecoveryStriped is tracked HDFS-12437 . Rest are passing locally.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks for the updated patch Brahma Reddy Battula, +1 pending the following nit and jenkins test:

        323       String  MIN_REPLICATION = PREFIX + "min-replication";
        324	  short  REPLICATION_DEFAULT = 0;
        

        The config and default value variable names are not consistent. Suggest to change the default value to MIN_REPLICATION_DEFAULT.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks for the updated patch Brahma Reddy Battula , +1 pending the following nit and jenkins test: 323 String MIN_REPLICATION = PREFIX + "min-replication" ; 324 short REPLICATION_DEFAULT = 0; The config and default value variable names are not consistent. Suggest to change the default value to MIN_REPLICATION_DEFAULT .
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch to address above comment.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to address above comment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 13m 9s trunk passed
        +1 compile 1m 32s trunk passed
        +1 checkstyle 0m 42s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 findbugs 3m 10s trunk passed
        +1 javadoc 1m 4s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 24s the patch passed
        +1 compile 1m 27s the patch passed
        +1 javac 1m 27s the patch passed
        -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 7 new + 156 unchanged - 0 fixed = 163 total (was 156)
        +1 mvnsite 1m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 21s the patch passed
        +1 javadoc 0m 58s the patch passed
              Other Tests
        +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
        -1 unit 87m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        121m 43s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.server.namenode.TestReencryptionWithKMS
          hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887692/HDFS-11799-008.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 7e86f3f0216e 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a2dcba1
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21196/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21196/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21196/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21196/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 13m 9s trunk passed +1 compile 1m 32s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 32s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 1m 4s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 1m 27s the patch passed +1 javac 1m 27s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 7 new + 156 unchanged - 0 fixed = 163 total (was 156) +1 mvnsite 1m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 21s the patch passed +1 javadoc 0m 58s the patch passed       Other Tests +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 87m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 121m 43s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887692/HDFS-11799-008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7e86f3f0216e 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a2dcba1 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21196/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21196/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21196/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21196/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Test Failures are unrelated. Yongjun Zhang can you take look now..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Test Failures are unrelated. Yongjun Zhang can you take look now..?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch to fix the checkstyle,I ignored this,yongjun thanks for reminding.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to fix the checkstyle,I ignored this,yongjun thanks for reminding.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Many thanks Brahma Reddy Battula. +1 pending jenkins.

        Show
        yzhangal Yongjun Zhang added a comment - Many thanks Brahma Reddy Battula . +1 pending jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 29s Maven dependency ordering for branch
        -1 mvninstall 13m 29s root in trunk failed.
        +1 compile 1m 28s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 findbugs 3m 4s trunk passed
        +1 javadoc 1m 5s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 21s the patch passed
        +1 compile 1m 24s the patch passed
        +1 javac 1m 24s the patch passed
        -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 156 unchanged - 0 fixed = 157 total (was 156)
        +1 mvnsite 1m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 47s the patch passed
        +1 javadoc 1m 4s the patch passed
              Other Tests
        +1 unit 1m 19s hadoop-hdfs-client in the patch passed.
        -1 unit 105m 32s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        140m 47s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestFileCorruption
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-11799
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887826/HDFS-11799-009.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux f414e3ab4abe 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2018538
        Default Java 1.8.0_144
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21206/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21206/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 29s Maven dependency ordering for branch -1 mvninstall 13m 29s root in trunk failed. +1 compile 1m 28s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 32s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 5s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 156 unchanged - 0 fixed = 157 total (was 156) +1 mvnsite 1m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 4s the patch passed       Other Tests +1 unit 1m 19s hadoop-hdfs-client in the patch passed. -1 unit 105m 32s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 140m 47s Reason Tests Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestFileCorruption Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-11799 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887826/HDFS-11799-009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux f414e3ab4abe 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2018538 Default Java 1.8.0_144 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21206/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21206/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21206/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        testfailures are unrelated. will commit shortly.

        Show
        brahmareddy Brahma Reddy Battula added a comment - testfailures are unrelated. will commit shortly.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12916 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12916/)
        HDFS-11799. Introduce a config to allow setting up write pipeline with (brahma: rev fda1221c55101d97ac62e1ee4e3ddf9a915d5363)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplaceDatanodeFailureReplication.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12916 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12916/ ) HDFS-11799 . Introduce a config to allow setting up write pipeline with (brahma: rev fda1221c55101d97ac62e1ee4e3ddf9a915d5363) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplaceDatanodeFailureReplication.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Committed to trunk,branch-3.0,branch-2 and branch-2.8. Yongjun Zhang thanks a lot for continuous review. Resolved minor conflicts for branch-2 and branch-2.8 and ran the testcase locally.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Committed to trunk , branch-3.0 , branch-2 and branch-2.8 . Yongjun Zhang thanks a lot for continuous review. Resolved minor conflicts for branch-2 and branch-2.8 and ran the testcase locally.

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            yzhangal Yongjun Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development