Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11856

Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates

    Details

      Description

      During rolling upgrade if the DN gets restarted, then it will send special OOB_RESTART status to all streams opened for write.
      1. Local clients will wait for 30 seconds to datanode to come back.
      2. Remote clients will consider these nodes as bad nodes and continue with pipeline recoveries and write. These restarted nodes will be considered as bad, and will be excluded for lifetime of stream.

      In case of small cluster, where total nodes itself is 3, each time a remote node restarts for upgrade, it will be excluded.
      So a stream writing to 3 nodes initial, will end-up writing to only one node at the end, there are no other nodes to replace.

      1. HDFS-11856-01.patch
        29 kB
        Vinayakumar B
      2. HDFS-11856-02.branch-2.patch
        30 kB
        Kihwal Lee
      3. HDFS-11856-02.patch
        29 kB
        Vinayakumar B
      4. HDFS-11856-branch-2.7-02.patch
        29 kB
        Vinayakumar B
      5. HDFS-11856-branch-2.8-02.patch
        30 kB
        Vinayakumar B
      6. HDFS-11856-branch-2-02.patch
        30 kB
        Vinayakumar B

        Activity

        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks a lot Kihwal Lee for reviews and commit.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks a lot Kihwal Lee for reviews and commit.
        Hide
        jzhuge John Zhuge added a comment -

        Thanks Vinayakumar B.

        Show
        jzhuge John Zhuge added a comment - Thanks Vinayakumar B .
        Hide
        vinayrpet Vinayakumar B added a comment -

        Will this patch help clusters with more than 3 DNs?

        Yes, this should work there as well. Only prerequisite is, OOB_RESTART flags should reach to client from upgrading node.

        Show
        vinayrpet Vinayakumar B added a comment - Will this patch help clusters with more than 3 DNs? Yes, this should work there as well. Only prerequisite is, OOB_RESTART flags should reach to client from upgrading node.
        Hide
        jzhuge John Zhuge added a comment -

        Vinayakumar B and Kihwal Lee Will this patch help clusters with more than 3 DNs? We saw HBase RegionServer occasional crashing with DamagedWALException after the following pipeline recovery failure:

        java.io.IOException: All datanodes DatanodeInfoWithStorage[x.x.x.x:20002,DS-uuid,DISK] are bad. Aborting...
                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
        
        Show
        jzhuge John Zhuge added a comment - Vinayakumar B and Kihwal Lee Will this patch help clusters with more than 3 DNs? We saw HBase RegionServer occasional crashing with DamagedWALException after the following pipeline recovery failure: java.io.IOException: All datanodes DatanodeInfoWithStorage[x.x.x.x:20002,DS-uuid,DISK] are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
        Hide
        kihwal Kihwal Lee added a comment -

        +1 for the branch-2.7 patch. I've just committed it.

        Show
        kihwal Kihwal Lee added a comment - +1 for the branch-2.7 patch. I've just committed it.
        Hide
        kihwal Kihwal Lee added a comment -

        branch-2/branch-2.8 precommit failure is being handled in HADOOP-14474.

        Show
        kihwal Kihwal Lee added a comment - branch-2/branch-2.8 precommit failure is being handled in HADOOP-14474 .
        Hide
        kihwal Kihwal Lee added a comment -

        +1 for the branch-2 patch. Need more time to look at the branch-2.7 one.

        Show
        kihwal Kihwal Lee added a comment - +1 for the branch-2 patch. Need more time to look at the branch-2.7 one.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 0m 32s Maven dependency ordering for branch
        +1 mvninstall 7m 19s branch-2 passed
        +1 compile 1m 32s branch-2 passed with JDK v1.8.0_131
        +1 compile 1m 31s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 40s branch-2 passed
        +1 mvnsite 1m 34s branch-2 passed
        +1 mvneclipse 0m 28s branch-2 passed
        +1 findbugs 3m 52s branch-2 passed
        +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 1m 28s branch-2 passed with JDK v1.7.0_131
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 16s the patch passed
        +1 compile 1m 27s the patch passed with JDK v1.8.0_131
        +1 javac 1m 27s the patch passed
        +1 compile 1m 38s the patch passed with JDK v1.7.0_131
        +1 javac 1m 38s the patch passed
        -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 4 new + 365 unchanged - 4 fixed = 369 total (was 369)
        +1 mvnsite 1m 29s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 33s the patch passed
        +1 javadoc 1m 1s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 28s the patch passed with JDK v1.7.0_131
        +1 unit 1m 12s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131.
        -1 unit 68m 37s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        166m 10s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.tools.TestDFSZKFailoverController
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
        JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:8515d35
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870084/HDFS-11856-02.branch-2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 74ad07e76ae0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / d36e64b
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19658/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19658/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19658/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19658/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 32s Maven dependency ordering for branch +1 mvninstall 7m 19s branch-2 passed +1 compile 1m 32s branch-2 passed with JDK v1.8.0_131 +1 compile 1m 31s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 40s branch-2 passed +1 mvnsite 1m 34s branch-2 passed +1 mvneclipse 0m 28s branch-2 passed +1 findbugs 3m 52s branch-2 passed +1 javadoc 1m 1s branch-2 passed with JDK v1.8.0_131 +1 javadoc 1m 28s branch-2 passed with JDK v1.7.0_131 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 27s the patch passed with JDK v1.8.0_131 +1 javac 1m 27s the patch passed +1 compile 1m 38s the patch passed with JDK v1.7.0_131 +1 javac 1m 38s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 4 new + 365 unchanged - 4 fixed = 369 total (was 369) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 33s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_131 +1 unit 1m 12s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131. -1 unit 68m 37s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 166m 10s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.tools.TestDFSZKFailoverController JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:8515d35 JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870084/HDFS-11856-02.branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74ad07e76ae0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d36e64b Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19658/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19658/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19658/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19658/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 11m 4s Docker failed to build yetus/hadoop:8515d35.



        Subsystem Report/Notes
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870084/HDFS-11856-02.branch-2.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19633/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 11m 4s Docker failed to build yetus/hadoop:8515d35. Subsystem Report/Notes JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870084/HDFS-11856-02.branch-2.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19633/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        Reattaching the branch-2 patch with a different name to kick the pre-commit.

        Show
        kihwal Kihwal Lee added a comment - Reattaching the branch-2 patch with a different name to kick the pre-commit.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 23s Docker failed to build yetus/hadoop:c420dfe.



        Subsystem Report/Notes
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870007/HDFS-11856-branch-2.7-02.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19627/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 23s Docker failed to build yetus/hadoop:c420dfe. Subsystem Report/Notes JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870007/HDFS-11856-branch-2.7-02.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19627/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        branch-2.7 patch attached.

        Show
        vinayrpet Vinayakumar B added a comment - branch-2.7 patch attached.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 4m 15s Docker failed to build yetus/hadoop:5970e82.



        Subsystem Report/Notes
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869999/HDFS-11856-branch-2.8-02.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19626/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 4m 15s Docker failed to build yetus/hadoop:5970e82. Subsystem Report/Notes JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869999/HDFS-11856-branch-2.8-02.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19626/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        patch for branch-2.8

        Show
        vinayrpet Vinayakumar B added a comment - patch for branch-2.8
        Hide
        vinayrpet Vinayakumar B added a comment -

        Attaching patch for branch-2.
        Others will attach soon.

        Show
        vinayrpet Vinayakumar B added a comment - Attaching patch for branch-2. Others will attach soon.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11783/)
        HDFS-11856. Ability to re-add Upgrading Nodes to pipeline for future (kihwal: rev 29b7df960fc3d0a7d1416225c3106c7d4222f0ca)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClientFaultInjector.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11783/ ) HDFS-11856 . Ability to re-add Upgrading Nodes to pipeline for future (kihwal: rev 29b7df960fc3d0a7d1416225c3106c7d4222f0ca) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClientFaultInjector.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        kihwal Kihwal Lee added a comment -

        Please provide patch for branch-2 and others.

        Show
        kihwal Kihwal Lee added a comment - Please provide patch for branch-2 and others.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 the patch looks good.

        Show
        kihwal Kihwal Lee added a comment - +1 the patch looks good.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 0m 34s Maven dependency ordering for branch
        +1 mvninstall 14m 8s trunk passed
        +1 compile 1m 48s trunk passed
        +1 checkstyle 0m 47s trunk passed
        +1 mvnsite 1m 47s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 3m 33s trunk passed
        +1 javadoc 1m 7s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 1m 47s the patch passed
        +1 javac 1m 47s the patch passed
        -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 4 new + 339 unchanged - 2 fixed = 343 total (was 341)
        +1 mvnsite 1m 36s the patch passed
        +1 mvneclipse 0m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 44s the patch passed
        +1 javadoc 1m 5s the patch passed
        +1 unit 1m 17s hadoop-hdfs-client in the patch passed.
        -1 unit 72m 18s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        111m 12s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869663/HDFS-11856-02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ba0c21df7c57 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a62be38
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19586/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19586/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19586/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19586/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 34s Maven dependency ordering for branch +1 mvninstall 14m 8s trunk passed +1 compile 1m 48s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 47s the patch passed +1 javac 1m 47s the patch passed -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 4 new + 339 unchanged - 2 fixed = 343 total (was 341) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 44s the patch passed +1 javadoc 1m 5s the patch passed +1 unit 1m 17s hadoop-hdfs-client in the patch passed. -1 unit 72m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 111m 12s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869663/HDFS-11856-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ba0c21df7c57 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a62be38 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19586/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19586/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19586/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19586/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Fixed TestWriteToReplica

        Show
        vinayrpet Vinayakumar B added a comment - Fixed TestWriteToReplica
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 13m 55s trunk passed
        +1 compile 1m 28s trunk passed
        +1 checkstyle 0m 40s trunk passed
        +1 mvnsite 1m 29s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 29s trunk passed
        +1 javadoc 1m 8s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 1m 28s the patch passed
        +1 javac 1m 28s the patch passed
        -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 3 new + 339 unchanged - 2 fixed = 342 total (was 341)
        +1 mvnsite 1m 31s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 49s the patch passed
        +1 javadoc 0m 58s the patch passed
        +1 unit 1m 21s hadoop-hdfs-client in the patch passed.
        -1 unit 82m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        119m 20s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869631/HDFS-11856-01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 66bb97c8e5b2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a62be38
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19579/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19579/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19579/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19579/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 13m 55s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 1m 28s the patch passed +1 javac 1m 28s the patch passed -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 3 new + 339 unchanged - 2 fixed = 342 total (was 341) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 49s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 82m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 119m 20s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869631/HDFS-11856-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 66bb97c8e5b2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a62be38 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19579/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19579/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19579/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19579/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Attached patch
        please review

        Show
        vinayrpet Vinayakumar B added a comment - Attached patch please review
        Hide
        vinayrpet Vinayakumar B added a comment -

        Solution:
        1. Consider the Non-Local restarting nodes as BAD for current pipeline update. Remove from bad nodes once the successful pipeline update happens.
        2. Allow datanode to accept replica transfers, if same replica with old genstamp exists already on node, irrespective of state of existing replica.

        Show
        vinayrpet Vinayakumar B added a comment - Solution: 1. Consider the Non-Local restarting nodes as BAD for current pipeline update. Remove from bad nodes once the successful pipeline update happens. 2. Allow datanode to accept replica transfers, if same replica with old genstamp exists already on node, irrespective of state of existing replica.

          People

          • Assignee:
            vinayrpet Vinayakumar B
            Reporter:
            vinayrpet Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development