Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11164

Mover should avoid unnecessary retries if the block is pinned

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: balancer & mover
    • Labels:
      None

      Description

      When mover is trying to move a pinned block to another datanode, it will internally hits the following IOException and mark the block movement as failure. Since the Mover has dfs.mover.retry.max.attempts configs, it will continue moving this block until it reaches retryMaxAttempts. If the block movement failure(s) are only due to block pinning, then retry is unnecessary. The idea of this jira is to avoid retry attempts of pinned blocks as they won't be able to move to a different node.

      2016-11-22 10:56:10,537 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
      java.io.IOException: Got error, status=ERROR, status message opReplaceBlock BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received exception java.io.IOException: Got error, status=ERROR, status message Not able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from /127.0.0.1:19501, reportedBlock move is failed
      	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
      	at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
      	at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
      	at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
      	at org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      1. HDFS-11164-00.patch
        23 kB
        Rakesh R
      2. HDFS-11164-01.patch
        23 kB
        Rakesh R
      3. HDFS-11164-02.patch
        32 kB
        Rakesh R
      4. HDFS-11164-03.patch
        32 kB
        Rakesh R

        Issue Links

          Activity

          Hide
          rakeshr Rakesh R added a comment -

          Presently DataXceiver#copyBlock block is having the datanode.data.getPinning(block) checks and throwing generic ERROR code to the caller. Since this err code is too generic, it would be difficult to differentiate the block pinned errors from other IOExceptions. One idea that comes in my mind is to introduce new error code say ERROR_BLOCK_PINNED represents the block pinning errors. That way, block pinning errors can be easily identified by other layers(mover) and do the necessary logic.

          DataXceiver.java
          
              if (datanode.data.getPinning(block)) {
                String msg = "Not able to copy block " + block.getBlockId() + " " +
                    "to " + peer.getRemoteAddressString() + " because it's pinned ";
                LOG.info(msg);
                sendResponse(ERROR_BLOCK_PINNED, msg);
                return;
              }
          
          Show
          rakeshr Rakesh R added a comment - Presently DataXceiver#copyBlock block is having the datanode.data.getPinning(block) checks and throwing generic ERROR code to the caller. Since this err code is too generic, it would be difficult to differentiate the block pinned errors from other IOExceptions. One idea that comes in my mind is to introduce new error code say ERROR_BLOCK_PINNED represents the block pinning errors. That way, block pinning errors can be easily identified by other layers(mover) and do the necessary logic. DataXceiver.java if (datanode.data.getPinning(block)) { String msg = "Not able to copy block " + block.getBlockId() + " " + "to " + peer.getRemoteAddressString() + " because it's pinned " ; LOG.info(msg); sendResponse(ERROR_BLOCK_PINNED, msg); return ; }
          Hide
          rakeshr Rakesh R added a comment -

          Attached an initial attempt by introducing ERROR_BLOCK_PINNED status code. Here the Mover tool will check that the block movement failure(s) are only due to block pinning. If yes, just mark as failed and exit without any retries.

          Show
          rakeshr Rakesh R added a comment - Attached an initial attempt by introducing ERROR_BLOCK_PINNED status code. Here the Mover tool will check that the block movement failure(s) are only due to block pinning. If yes, just mark as failed and exit without any retries.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 1m 20s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 1m 2s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 1m 24s the patch passed
          +1 cc 1m 24s the patch passed
          +1 javac 1m 24s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 192 unchanged - 0 fixed = 194 total (was 192)
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 50s the patch passed
          +1 javadoc 1m 2s the patch passed
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          -1 unit 67m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          95m 40s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11164
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840025/HDFS-11164-00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 277d562c116d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a926f89
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17632/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17632/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17632/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 1m 20s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 24s the patch passed +1 cc 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 192 unchanged - 0 fixed = 194 total (was 192) +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 50s the patch passed +1 javadoc 1m 2s the patch passed +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 67m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 95m 40s Reason Tests Failed junit tests hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840025/HDFS-11164-00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 277d562c116d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a926f89 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17632/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17632/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17632/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 8m 33s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 31s the patch passed
          +1 cc 1m 31s the patch passed
          +1 javac 1m 31s the patch passed
          +1 checkstyle 0m 34s the patch passed
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 1s the patch passed
          +1 javadoc 1m 10s the patch passed
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 123m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          155m 30s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.TestBlocksScheduledCounter
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11164
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840063/HDFS-11164-01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux abe1c66e33cd 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / afcf8d3
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17635/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17635/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17635/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 8m 33s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 31s the patch passed +1 cc 1m 31s the patch passed +1 javac 1m 31s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 1s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 123m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 155m 30s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840063/HDFS-11164-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux abe1c66e33cd 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / afcf8d3 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17635/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17635/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17635/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Rakesh R, Thanks for reporting the issue.
          Adding extra error code make sense to me.

          But the current patch may not solve for avoiding unnecessary retries issue I think.
          Please check the following cases, correct me if I am wrong.

           // Check that the block movement failure(s) are only due to block pinning.
                // If yes, just mark as failed and exit without retries.
                if(!hasFailed && hasBlockPinningFailure){
                  hasFailed = hasBlockPinningFailure;
                  result.setRetryFailed();
                } else if (hasFailed && !hasSuccess) {
                  if (retryCount.get() == retryMaxAttempts) {
                    result.setRetryFailed();
                    LOG.error("Failed to move some block's after "
                        + retryMaxAttempts + " retries.");
                    return result;
                  } else {
                    retryCount.incrementAndGet();
                  }
                } else {
                  // Reset retry count if no failure.
                  retryCount.set(0);
                }
                result.updateHasRemaining(hasFailed);
                return result;
          

          Here !hasFailed && hasBlockPinningFailure case is targeting for only pinned failure and no normal failures right? if so, when there are normal failures and pinned failures together, it will still retry?
          If it retries, it may scan that paths again and try to move even they are pinned blocks.
          We may need to think this in a bit different way than node level failures I think.
          One thought is, Failed blocks due to pinned can be stored separately and when retry happens and if blocks exist in failedDueToPinned list, then skip to add them into PendingMoves? Just a thought, we need to check the feasibility.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Rakesh R , Thanks for reporting the issue. Adding extra error code make sense to me. But the current patch may not solve for avoiding unnecessary retries issue I think. Please check the following cases, correct me if I am wrong. // Check that the block movement failure(s) are only due to block pinning. // If yes, just mark as failed and exit without retries. if (!hasFailed && hasBlockPinningFailure){ hasFailed = hasBlockPinningFailure; result.setRetryFailed(); } else if (hasFailed && !hasSuccess) { if (retryCount.get() == retryMaxAttempts) { result.setRetryFailed(); LOG.error( "Failed to move some block's after " + retryMaxAttempts + " retries." ); return result; } else { retryCount.incrementAndGet(); } } else { // Reset retry count if no failure. retryCount.set(0); } result.updateHasRemaining(hasFailed); return result; Here !hasFailed && hasBlockPinningFailure case is targeting for only pinned failure and no normal failures right? if so, when there are normal failures and pinned failures together, it will still retry? If it retries, it may scan that paths again and try to move even they are pinned blocks. We may need to think this in a bit different way than node level failures I think. One thought is, Failed blocks due to pinned can be stored separately and when retry happens and if blocks exist in failedDueToPinned list, then skip to add them into PendingMoves? Just a thought, we need to check the feasibility.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Uma Maheswara Rao G for the comments.

          One thought is, Failed blocks due to pinned can be stored separately and when retry happens and if blocks exist in failedDueToPinned list, then skip to add them into PendingMoves? Just a thought, we need to check the feasibility.

          Yes, we can maintain a datastructure to hold the blockId vs pinnedBlkLocations, there could be multiple favored/pinned nodes for a "blockId". Let me try implementing this excluded logic. In worst case, I could see a case of millions of pinned block entries and this data structure will consume more memory throughout the lifetime of Mover tool. Do we need to introduce any configuration parameter to limit the size of this data structure.

          int LIMIT = // configured value, default to 10K.
          Map<Long, Set<DatanodeInfo>> excludedPinnedBlocks = new HashMap<>(LIMIT);
          
          Show
          rakeshr Rakesh R added a comment - Thanks Uma Maheswara Rao G for the comments. One thought is, Failed blocks due to pinned can be stored separately and when retry happens and if blocks exist in failedDueToPinned list, then skip to add them into PendingMoves? Just a thought, we need to check the feasibility. Yes, we can maintain a datastructure to hold the blockId vs pinnedBlkLocations , there could be multiple favored/pinned nodes for a "blockId". Let me try implementing this excluded logic. In worst case, I could see a case of millions of pinned block entries and this data structure will consume more memory throughout the lifetime of Mover tool. Do we need to introduce any configuration parameter to limit the size of this data structure. int LIMIT = // configured value, default to 10K. Map< Long , Set<DatanodeInfo>> excludedPinnedBlocks = new HashMap<>(LIMIT);
          Hide
          rakeshr Rakesh R added a comment -

          Attached another patch. Here, I've tried an attempt by maintaining Map<Long, Set<DatanodeInfo>> excludedPinnedBlocks; details at the Mover.

          Show
          rakeshr Rakesh R added a comment - Attached another patch. Here, I've tried an attempt by maintaining Map<Long, Set<DatanodeInfo>> excludedPinnedBlocks; details at the Mover.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 7m 29s trunk passed
          +1 compile 1m 28s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 3m 14s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 1m 23s the patch passed
          +1 cc 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 21s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 94m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          122m 57s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11164
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840590/HDFS-11164-02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux b36280c5c846 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c06114d
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17670/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17670/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17670/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 7m 29s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 1m 23s the patch passed +1 cc 1m 23s the patch passed +1 javac 1m 23s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 21s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 94m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 122m 57s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840590/HDFS-11164-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux b36280c5c846 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c06114d Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17670/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17670/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17670/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 8m 59s trunk passed
          +1 compile 1m 32s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 40s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 1m 35s the patch passed
          +1 cc 1m 35s the patch passed
          +1 javac 1m 35s the patch passed
          +1 checkstyle 0m 32s the patch passed
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 53s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed.
          +1 unit 71m 59s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          103m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11164
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840815/HDFS-11164-03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux ca2e5e2a7059 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67d9f28
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17693/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17693/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 8m 59s trunk passed +1 compile 1m 32s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 40s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 1m 35s the patch passed +1 cc 1m 35s the patch passed +1 javac 1m 35s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 53s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 1m 5s hadoop-hdfs-client in the patch passed. +1 unit 71m 59s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 103m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840815/HDFS-11164-03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ca2e5e2a7059 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67d9f28 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17693/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17693/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Rakesh R Overall patch looks good to me. +1

          Surendra Singh Lilhore since you tested mover scenarios before, do you mind checking this patch with your clusters whether its effecting any of your scenarios?

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Rakesh R Overall patch looks good to me. +1 Surendra Singh Lilhore since you tested mover scenarios before, do you mind checking this patch with your clusters whether its effecting any of your scenarios?
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Sure Uma Maheswara Rao G, I will verify the patch in my test cluster..
          I will update the result tomorrow..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Sure Uma Maheswara Rao G , I will verify the patch in my test cluster.. I will update the result tomorrow..
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          I tested and its working fine.

          I have one suggestion here.
          I feel pinned block information should be given by namenode (Dananode will send pinned block info to namenode in block report). So we can skip first try also to move pinned blocks..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - I tested and its working fine. I have one suggestion here. I feel pinned block information should be given by namenode (Dananode will send pinned block info to namenode in block report). So we can skip first try also to move pinned blocks..
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Uma Maheswara Rao G for the reviews.

          Show
          rakeshr Rakesh R added a comment - Thank you Uma Maheswara Rao G for the reviews.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks a lot Surendra Singh Lilhore for the test feedback.

          IMHO, this information is not required to place in NN and the first failure is OK considering the additional cost of maintaining the block pinned info at the NN server and again this has to be synced up during DN restart via block report etc. Also, I've gone through the HDFS-6133 comments and I could see some interesting discussion about the design of block pinning. Do you agree to go ahead with the proposed approach in the patch?

          Show
          rakeshr Rakesh R added a comment - Thanks a lot Surendra Singh Lilhore for the test feedback. IMHO, this information is not required to place in NN and the first failure is OK considering the additional cost of maintaining the block pinned info at the NN server and again this has to be synced up during DN restart via block report etc. Also, I've gone through the HDFS-6133 comments and I could see some interesting discussion about the design of block pinning . Do you agree to go ahead with the proposed approach in the patch?
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Thanks a lot, Surendra Singh Lilhore for verification and confirming.

          Thank you for suggesting ideas. In general its a nice idea, but the fact we should consider is, keeping less maintenance work at NN. Since this is not critical namespace info/block info, its ok to leave this info to DN.

          I will go ahead to commit this patch!

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Thanks a lot, Surendra Singh Lilhore for verification and confirming. Thank you for suggesting ideas. In general its a nice idea, but the fact we should consider is, keeping less maintenance work at NN. Since this is not critical namespace info/block info, its ok to leave this info to DN. I will go ahead to commit this patch!
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          I have just committed this to trunk.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - I have just committed this to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10992 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10992/)
          HDFS-11164: Mover should avoid unnecessary retries if the block is (uma.gangumalla: rev e24a923db50879f7dbe5d2afac0e6757089fb07d)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/BlockPinningException.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10992 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10992/ ) HDFS-11164 : Mover should avoid unnecessary retries if the block is (uma.gangumalla: rev e24a923db50879f7dbe5d2afac0e6757089fb07d) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/BlockPinningException.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java

            People

            • Assignee:
              rakeshr Rakesh R
              Reporter:
              rakeshr Rakesh R
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development