Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8693

refreshNamenodes does not support adding a new standby to a running DN

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: datanode, ha
    • Labels:
      None

      Description

      I tried to run the following command on a Hadoop 2.6.0 cluster with HA support

      $ hdfs dfsadmin -refreshNamenodes datanode-host:port

      to refresh name nodes on data nodes after I replaced one name node with a new one so that I don't need to restart the data nodes. However, I got the following error:

      refreshNamenodes: HA does not currently support adding a new standby to a running DN. Please do a rolling restart of DNs to reconfigure the list of NNs.

      I checked the 2.6.0 code and the error was thrown by the following code snippet, which led me to this JIRA.

      void refreshNNList(ArrayList<InetSocketAddress> addrs) throws IOException

      { Set<InetSocketAddress> oldAddrs = Sets.newHashSet(); for (BPServiceActor actor : bpServices) { oldAddrs.add(actor.getNNSocketAddress()); }

      Set<InetSocketAddress> newAddrs = Sets.newHashSet(addrs);
      if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty())

      { // Keep things simple for now -- we can implement this at a later date. throw new IOException( "HA does not currently support adding a new standby to a running DN. " + "Please do a rolling restart of DNs to reconfigure the list of NNs."); }

      }

      Looks like this the refreshNameNodes command is an uncompleted feature.

      Unfortunately, the new name node on a replacement is critical for auto provisioning a hadoop cluster with HDFS HA support. Without this support, the HA feature could not really be used. I also observed that the new standby name node on the replacement instance could stuck in safe mode because no data nodes check in with it. Even with a rolling restart, it may take quite some time to restart all data nodes if we have a big cluster, for example, with 4000 data nodes, let alone restarting DN is way too intrusive and it is not a preferable operation in production. It also increases the chance for a double failure because the standby name node is not really ready for a failover in the case that the current active name node fails.

      1. HDFS-8693.1.patch
        5 kB
        Ajith S
      2. HDFS-8693.02.patch
        5 kB
        Ajith S
      3. HDFS-8693.03.patch
        5 kB
        Ajith S

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          If namenode address changes, every clients including datanodes need to be updated. This is not really feasible if you have long-running services and clients, so most people configure the name node service addresses to be DNS aliases. When a node needs to be swapped, it is a matter of configuring an ethernet interface alias.

          Show
          kihwal Kihwal Lee added a comment - If namenode address changes, every clients including datanodes need to be updated. This is not really feasible if you have long-running services and clients, so most people configure the name node service addresses to be DNS aliases. When a node needs to be swapped, it is a matter of configuring an ethernet interface alias.
          Hide
          kihwal Kihwal Lee added a comment -

          I do agree that refreshNameNodes needs to be fixed. This command does not work for federated HA clusters. Also, if one service actor thread shuts down in HA, there is no way to start it up again without restarting the datanode. The datanode should shutdown in this case, or refreshNamenodes should be fixed to work with HA.

          Show
          kihwal Kihwal Lee added a comment - I do agree that refreshNameNodes needs to be fixed. This command does not work for federated HA clusters. Also, if one service actor thread shuts down in HA, there is no way to start it up again without restarting the datanode. The datanode should shutdown in this case, or refreshNamenodes should be fixed to work with HA.
          Hide
          john.jian.fang Jian Fang added a comment -

          Kihwal Lee, thank you for your comments. Unfortunately, DNS alias does not really work or comes with a very high maintenance cost in some scenarios when the cluster is not maintained by system admin, but by software such as the case of auto provisioning and managing Hadoop cluster in cloud. HDFS clients have the failover proxy configured to switch to one of the two pre-configured name node addresses. There are cases the new name node becomes active and the HDFS clients are broken, but we more concern about the data nodes themselves for a long running cluster. The refreshNameNodes needs to be fixed in this case.

          Show
          john.jian.fang Jian Fang added a comment - Kihwal Lee , thank you for your comments. Unfortunately, DNS alias does not really work or comes with a very high maintenance cost in some scenarios when the cluster is not maintained by system admin, but by software such as the case of auto provisioning and managing Hadoop cluster in cloud. HDFS clients have the failover proxy configured to switch to one of the two pre-configured name node addresses. There are cases the new name node becomes active and the HDFS clients are broken, but we more concern about the data nodes themselves for a long running cluster. The refreshNameNodes needs to be fixed in this case.
          Hide
          ajithshetty Ajith S added a comment -

          Hi Kihwal Lee

          I tested with federated HA cluster when adding a new nameservice, the command works. Is there any special scenario when you said it doesn't work for federated HA cluster.?

          Show
          ajithshetty Ajith S added a comment - Hi Kihwal Lee I tested with federated HA cluster when adding a new nameservice, the command works. Is there any special scenario when you said it doesn't work for federated HA cluster.?
          Hide
          ajithshetty Ajith S added a comment -

          Hi Jian Fang and Kihwal Lee

          Agreed, need to fix refreshNameNodes. In refreshNNList, can we just add a new NN actor and replace the old NN actor in block pool service.??
          I would like to work on this issue

          Show
          ajithshetty Ajith S added a comment - Hi Jian Fang and Kihwal Lee Agreed, need to fix refreshNameNodes. In refreshNNList, can we just add a new NN actor and replace the old NN actor in block pool service.?? I would like to work on this issue
          Hide
          ajithshetty Ajith S added a comment -

          Attaching Patch. Please review

          Show
          ajithshetty Ajith S added a comment - Attaching Patch. Please review
          Hide
          vinayrpet Vinayakumar B added a comment -

          Changes looks good.
          However, precommit has not run yet on this patch.
          Triggered one. Currently looks like jenkin nodes have some problem.
          Will wait for the QA reports.

          Kihwal Lee, You might be interested in this. This fixes the above issue you mentioned.

          Show
          vinayrpet Vinayakumar B added a comment - Changes looks good. However, precommit has not run yet on this patch. Triggered one. Currently looks like jenkin nodes have some problem. Will wait for the QA reports. Kihwal Lee , You might be interested in this. This fixes the above issue you mentioned.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HDFS-8693 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-8693
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796035/HDFS-8693.1.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17450/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HDFS-8693 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-8693 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796035/HDFS-8693.1.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17450/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajithshetty Ajith S added a comment -

          Attacing rebased patch. Please review

          Show
          ajithshetty Ajith S added a comment - Attacing rebased patch. Please review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 6s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 49 unchanged - 1 fixed = 50 total (was 50)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 80m 22s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          100m 8s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-8693
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 537154b6144a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b970446
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17452/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17452/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17452/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17452/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 6s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 49 unchanged - 1 fixed = 50 total (was 50) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 80m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 100m 8s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-8693 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 537154b6144a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b970446 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17452/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17452/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17452/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17452/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 1s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 49 unchanged - 1 fixed = 50 total (was 50)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 57m 46s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          76m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-8693
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5a127b773016 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f768955
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17454/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17454/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17454/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17454/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 1s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 49 unchanged - 1 fixed = 50 total (was 50) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 57m 46s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 76m 28s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-8693 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5a127b773016 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f768955 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17454/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17454/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17454/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17454/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhangchen Chen Zhang added a comment -

          Hi Vinayakumar B Ajith S, any progress on this issue?
          Supporting add a new standby is very useful for large cluster operation. When one of the machine running namenode is down and we have to add another new standby, restarting thousands of datanodes will take very long time. Once the active namenode is crushed during this time, whole cluster will not available.

          Show
          zhangchen Chen Zhang added a comment - Hi Vinayakumar B Ajith S , any progress on this issue? Supporting add a new standby is very useful for large cluster operation. When one of the machine running namenode is down and we have to add another new standby, restarting thousands of datanodes will take very long time. Once the active namenode is crushed during this time, whole cluster will not available.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 10s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51)
          +1 mvnsite 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 43s the patch passed
          -1 unit 95m 45s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          121m 31s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-8693
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b02065b0771b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0111711
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20034/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20034/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20034/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20034/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 10s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 53s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51) +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 43s the patch passed -1 unit 95m 45s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 121m 31s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-8693 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837756/HDFS-8693.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b02065b0771b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20034/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20034/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20034/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20034/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajithshetty Ajith S added a comment -

          Hi Chen Zhang
          I had uploaded the patch sometime back, let me try to rebase and address test case failure for this

          Show
          ajithshetty Ajith S added a comment - Hi Chen Zhang I had uploaded the patch sometime back, let me try to rebase and address test case failure for this
          Hide
          ajithshetty Ajith S added a comment -

          Will rebase and upload new

          Show
          ajithshetty Ajith S added a comment - Will rebase and upload new
          Hide
          ajithshetty Ajith S added a comment -

          Attached rebased patch

          Show
          ajithshetty Ajith S added a comment - Attached rebased patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 26s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 0m 46s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 58s the patch passed
          +1 javac 0m 58s the patch passed
          +1 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 51 unchanged - 1 fixed = 51 total (was 52)
          +1 mvnsite 1m 4s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 0m 43s the patch passed
                Other Tests
          -1 unit 78m 22s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          107m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-8693
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882813/HDFS-8693.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bb48ff496ea1 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7a82d7b
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20773/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20773/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20773/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20773/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 26s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 5s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 46s trunk passed       Patch Compile Tests +1 mvninstall 0m 59s the patch passed +1 compile 0m 58s the patch passed +1 javac 0m 58s the patch passed +1 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 51 unchanged - 1 fixed = 51 total (was 52) +1 mvnsite 1m 4s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 2m 8s the patch passed +1 javadoc 0m 43s the patch passed       Other Tests -1 unit 78m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 107m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-8693 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882813/HDFS-8693.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb48ff496ea1 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7a82d7b Default Java 1.8.0_144 findbugs v3.1.0-RC1 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20773/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20773/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20773/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20773/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              ajithshetty Ajith S
              Reporter:
              john.jian.fang Jian Fang
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development