Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This new dfsadmin command, evictWriters, stops active block writing activities on a data node. The affected writes will continue without the node after a write pipeline recovery. This is useful when data node decommissioning is blocked by slow writers. If issued against a non-decommissioing data node, all current writers will be stopped, but new write requests will continue to be served.
      Show
      This new dfsadmin command, evictWriters, stops active block writing activities on a data node. The affected writes will continue without the node after a write pipeline recovery. This is useful when data node decommissioning is blocked by slow writers. If issued against a non-decommissioing data node, all current writers will be stopped, but new write requests will continue to be served.

      Description

      It will be useful if there is a command to evict writers from a datanode. When a set of datanodes are being decommissioned, they can get blocked by slow writers at the end. It was rare in the old days since mapred jobs didn't last too long, but with many different types of apps running on today's YARN cluster, we are often see very long tail in datanode decommissioning.

      I propose a new dfsadmin command, evictWriters, to be added. I initially thought about having namenode automatically telling datanodes on decommissioning, but realized that having a command is more flexible. E.g. users can choose not to do this at all, choose when to evict writers, or whether to try multiple times for whatever reasons.

      1. HDFS-9945.patch
        18 kB
        Kihwal Lee
      2. HDFS-9945.v2.patch
        18 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9570 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9570/)
          HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal (epayne: rev aede8c10ecad4f2a8802a834e4bd0b8286cebade)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9570 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9570/ ) HDFS-9945 . Datanode command for evicting writers. Contributed by Kihwal (epayne: rev aede8c10ecad4f2a8802a834e4bd0b8286cebade) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Kihwal Lee, for providing this feature. The patch looks good to me.

          +1. I will commit to trunk, branch-2, and branch-2.8

          Show
          eepayne Eric Payne added a comment - Thanks, Kihwal Lee , for providing this feature. The patch looks good to me. +1. I will commit to trunk, branch-2, and branch-2.8
          Hide
          kihwal Kihwal Lee added a comment -

          HDFS-2043 TestHFlush
          HDFS-9780 TestRollingFileSystemSinkWithSecureHdfs
          HDFS-9950 TestDecommissioningStatus
          HDFS-10169 TestEditLog
          HDFS-9767 TestFileAppend
          HDFS-6532 TestCrcCorruption
          I will work on some of these.

          The two checkstyle warnings are about the existing method length being over 150 lines.

          Show
          kihwal Kihwal Lee added a comment - HDFS-2043 TestHFlush HDFS-9780 TestRollingFileSystemSinkWithSecureHdfs HDFS-9950 TestDecommissioningStatus HDFS-10169 TestEditLog HDFS-9767 TestFileAppend HDFS-6532 TestCrcCorruption I will work on some of these. The two checkstyle warnings are about the existing method length being over 150 lines.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 31s trunk passed
          +1 compile 1m 48s trunk passed with JDK v1.8.0_74
          +1 compile 1m 34s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 48s trunk passed
          +1 javadoc 1m 43s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 34s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 1m 50s the patch passed with JDK v1.8.0_74
          +1 cc 1m 50s the patch passed
          +1 javac 1m 50s the patch passed
          +1 compile 1m 36s the patch passed with JDK v1.7.0_95
          +1 cc 1m 36s the patch passed
          +1 javac 1m 36s the patch passed
          -1 checkstyle 0m 35s hadoop-hdfs-project: patch generated 2 new + 538 unchanged - 4 fixed = 540 total (was 542)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 28s the patch passed
          +1 javadoc 1m 48s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 21s the patch passed with JDK v1.7.0_95
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 85m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 79m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          208m 9s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793390/HDFS-9945.v2.patch
          JIRA Issue HDFS-9945
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux ff4eb2a03996 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bd5556b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14814/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14814/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 31s trunk passed +1 compile 1m 48s trunk passed with JDK v1.8.0_74 +1 compile 1m 34s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 43s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 34s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 1m 50s the patch passed with JDK v1.8.0_74 +1 cc 1m 50s the patch passed +1 javac 1m 50s the patch passed +1 compile 1m 36s the patch passed with JDK v1.7.0_95 +1 cc 1m 36s the patch passed +1 javac 1m 36s the patch passed -1 checkstyle 0m 35s hadoop-hdfs-project: patch generated 2 new + 538 unchanged - 4 fixed = 540 total (was 542) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 28s the patch passed +1 javadoc 1m 48s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 21s the patch passed with JDK v1.7.0_95 +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 85m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 79m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 208m 9s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.TestDecommissioningStatus JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestEditLog   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793390/HDFS-9945.v2.patch JIRA Issue HDFS-9945 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ff4eb2a03996 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bd5556b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14814/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14814/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14814/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Fixed the findbug warning and two checkstyle issues.

          Show
          kihwal Kihwal Lee added a comment - Fixed the findbug warning and two checkstyle issues.
          Hide
          kihwal Kihwal Lee added a comment -

          Pre-commit actually ran: https://builds.apache.org/job/PreCommit-HDFS-Build/14795/
          It didn't post the result probably because of the jira issue at that time.

          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 57s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author
                tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or
                modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 1m 18s trunk passed with JDK v1.8.0_74
          +1 compile 1m 21s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 37s trunk passed
          +1 javadoc 1m 25s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 1m 13s the patch passed with JDK v1.8.0_74
          +1 cc 1m 13s the patch passed
          +1 javac 1m 13s the patch passed
          +1 compile 1m 19s the patch passed with JDK v1.7.0_95
          +1 cc 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          -1 checkstyle 0m 31s hadoop-hdfs-project: patch generated 4
                new + 538 unchanged - 4 fixed = 542 total
                (was 542)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 11s hadoop-hdfs-project/hadoop-hdfs
                generated 1 new + 0 unchanged - 0 fixed =
                1 total (was 0)
          +1 javadoc 1m 23s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed
                with JDK v1.8.0_74.
          -1 unit 55m 54s hadoop-hdfs in the patch failed with JDK
                v1.8.0_74.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed
                with JDK v1.7.0_95.
          +1 unit 53m 36s hadoop-hdfs in the patch passed with JDK
                v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License
                warnings.
              149m 8s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Inconsistent synchronization of org.apache.hadoop.hdfs.server.datanode.DataXceiver.xceiver; locked 50% of time Unsynchronized access at DataXceiver.java:50% of time Unsynchronized access at DataXceiver.java:[line 222]
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.TestFileAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792899/HDFS-9945.patch
          JIRA Issue HDFS-9945
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux be2432938a57 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 658ee95
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14795/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14795/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org
          Show
          kihwal Kihwal Lee added a comment - Pre-commit actually ran: https://builds.apache.org/job/PreCommit-HDFS-Build/14795/ It didn't post the result probably because of the jira issue at that time. -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 57s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author       tags. +1 test4tests 0m 0s The patch appears to include 1 new or       modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 1m 18s trunk passed with JDK v1.8.0_74 +1 compile 1m 21s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 37s trunk passed +1 javadoc 1m 25s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 13s the patch passed with JDK v1.8.0_74 +1 cc 1m 13s the patch passed +1 javac 1m 13s the patch passed +1 compile 1m 19s the patch passed with JDK v1.7.0_95 +1 cc 1m 19s the patch passed +1 javac 1m 19s the patch passed -1 checkstyle 0m 31s hadoop-hdfs-project: patch generated 4       new + 538 unchanged - 4 fixed = 542 total       (was 542) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 11s hadoop-hdfs-project/hadoop-hdfs       generated 1 new + 0 unchanged - 0 fixed =       1 total (was 0) +1 javadoc 1m 23s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95 +1 unit 0m 49s hadoop-hdfs-client in the patch passed       with JDK v1.8.0_74. -1 unit 55m 54s hadoop-hdfs in the patch failed with JDK       v1.8.0_74. +1 unit 0m 56s hadoop-hdfs-client in the patch passed       with JDK v1.7.0_95. +1 unit 53m 36s hadoop-hdfs in the patch passed with JDK       v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License       warnings.     149m 8s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Inconsistent synchronization of org.apache.hadoop.hdfs.server.datanode.DataXceiver.xceiver; locked 50% of time Unsynchronized access at DataXceiver.java:50% of time Unsynchronized access at DataXceiver.java:[line 222] JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.TestFileAppend Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792899/HDFS-9945.patch JIRA Issue HDFS-9945 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux be2432938a57 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 658ee95 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14795/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14795/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14795/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org
          Hide
          kihwal Kihwal Lee added a comment -

          This patch adds a new dfsadmin command that talks to the specified datanode to stop all writers. Since it is a new method and feature, the risk of breaking existing code is low.

          The only notable change to the existing code is making replaceBlock() update the blockReceiver class variable instead of its own local variable. We want evictWriters() to also stop replications and balancer-related write activities. Because of this change, sendOOB() was updated to not send if isDatanode is true. Prior to this change, sendOOB() was not called at all when replaceBlock() is in progress, because blockReceiver, the class variable, was still null.

          Show
          kihwal Kihwal Lee added a comment - This patch adds a new dfsadmin command that talks to the specified datanode to stop all writers. Since it is a new method and feature, the risk of breaking existing code is low. The only notable change to the existing code is making replaceBlock() update the blockReceiver class variable instead of its own local variable. We want evictWriters() to also stop replications and balancer-related write activities. Because of this change, sendOOB() was updated to not send if isDatanode is true. Prior to this change, sendOOB() was not called at all when replaceBlock() is in progress, because blockReceiver , the class variable, was still null.

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development