Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9713

DataXceiver#copyBlock should return if block is pinned

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      in DataXceiver#copyBlock

        if (datanode.data.getPinning(block)) {
            String msg = "Not able to copy block " + block.getBlockId() + " " +
                "to " + peer.getRemoteAddressString() + " because it's pinned ";
            LOG.info(msg);
            sendResponse(ERROR, msg);
          }
      

      I think we should return back instead of proceeding to send block.as we already sent ERROR here.

      1. HDFS-9713-00.patch
        0.8 kB
        Uma Maheswara Rao G

        Activity

        Hide
        drankye Kai Zheng added a comment -

        Good catch Uma. I thought you're right.

        Show
        drankye Kai Zheng added a comment - Good catch Uma. I thought you're right.
        Hide
        linyiqun Yiqun Lin added a comment -

        That's a problem here.In DataXceiver#copyBlock:

        if (datanode.data.getPinning(block)) {
              String msg = "Not able to copy block " + block.getBlockId() + " " +
                  "to " + peer.getRemoteAddressString() + " because it's pinned ";
              LOG.info(msg);
              sendResponse(ERROR, msg);
            }
            
            if (!dataXceiverServer.balanceThrottler.acquire()) { // not able to start
              String msg = "Not able to copy block " + block.getBlockId() + " " +
                  "to " + peer.getRemoteAddressString() + " because threads " +
                  "quota is exceeded.";
              LOG.warn(msg);
              sendResponse(ERROR, msg);
              return;
            }
        

        One logic have returned, and the other is not. Uma Maheswara Rao G, this jira have not updated many days, you can assign to me if you have not time to do with it.

        Show
        linyiqun Yiqun Lin added a comment - That's a problem here.In DataXceiver#copyBlock: if (datanode.data.getPinning(block)) { String msg = "Not able to copy block " + block.getBlockId() + " " + "to " + peer.getRemoteAddressString() + " because it's pinned " ; LOG.info(msg); sendResponse(ERROR, msg); } if (!dataXceiverServer.balanceThrottler.acquire()) { // not able to start String msg = "Not able to copy block " + block.getBlockId() + " " + "to " + peer.getRemoteAddressString() + " because threads " + "quota is exceeded." ; LOG.warn(msg); sendResponse(ERROR, msg); return ; } One logic have returned, and the other is not. Uma Maheswara Rao G , this jira have not updated many days, you can assign to me if you have not time to do with it.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Here is a simple fix. This is straightforward change. I have verified it with TestBalancer.java, which is having a testcase for pinned blocks. So, even after return the current behavior should not change and it should work as is.

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hdfs.server.balancer.TestBalancer
        Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 322.695 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancer

        Results :

        Tests run: 31, Failures: 0, Errors: 0, Skipped: 0

        Kai Zheng can you please review.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Here is a simple fix. This is straightforward change. I have verified it with TestBalancer.java, which is having a testcase for pinned blocks. So, even after return the current behavior should not change and it should work as is. ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.balancer.TestBalancer Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 322.695 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancer Results : Tests run: 31, Failures: 0, Errors: 0, Skipped: 0 Kai Zheng can you please review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 44s trunk passed
        +1 compile 0m 41s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_66
        +1 javac 0m 37s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_91
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 5s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91
        -1 unit 58m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 53m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 27s Patch does not generate ASF License warnings.
        137m 7s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785954/HDFS-9713-00.patch
        JIRA Issue HDFS-9713
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7664749df5cd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / eb2fb94
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14357/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14357/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 45s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 5s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91 -1 unit 58m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 53m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 137m 7s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785954/HDFS-9713-00.patch JIRA Issue HDFS-9713 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7664749df5cd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / eb2fb94 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14357/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14357/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14357/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        +1, good catch. Thanks Uma.

        Show
        hitliuyi Yi Liu added a comment - +1, good catch. Thanks Uma.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        I have just committed this to trunk, branch2 and branch-2.8

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - I have just committed this to trunk, branch2 and branch-2.8
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9257/)
        HDFS-9713. DataXceiver#copyBlock should return if block is pinned. (uma.gangumalla: rev aea31eee78a287f251447686ba3f5bbcfe9c60a3)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9257/ ) HDFS-9713 . DataXceiver#copyBlock should return if block is pinned. (uma.gangumalla: rev aea31eee78a287f251447686ba3f5bbcfe9c60a3) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            umamaheswararao Uma Maheswara Rao G
            Reporter:
            umamaheswararao Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development