Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6147

New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      New blocks scanning will be delayed if old blocks deleted after datanode restart.

      Steps:
      1. Write some blocks and wait till all scans over
      2. Restart the datanode
      3. Delete some of the blocks
      4. Write new blocks which are less in size compared to deleted blocks.

      Problem:
      BlockPoolSliceScanner#updateBytesToScan(..) updates bytesLeft based on following comparison

         if (lastScanTime < currentPeriodStart) {
            bytesLeft += len;
          }

      But in BlockPoolSliceScanner#assignInitialVerificationTimes() bytesLeft decremented using below comparison

      if (now - entry.verificationTime < scanPeriod) {

      Hence when the old blocks are deleted bytesLeft going negative.
      new blocks will not be scanned until it becomes positive again.

      So in both places verificationtime should be compared against scanperiod.

      1. HDFS-6147.patch
        14 kB
        Vinayakumar B
      2. HDFS-6147.patch
        14 kB
        Vinayakumar B
      3. HDFS-6147.patch
        13 kB
        Vinayakumar B
      4. HDFS-6147.patch
        13 kB
        Vinayakumar B
      5. HDFS-6147.patch
        13 kB
        Vinayakumar B
      6. HDFS-6147.patch
        8 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12636312/HDFS-6147.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12636312/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//console This message is automatically generated.
          Hide
          Vinayakumar B added a comment -

          Attaching the patch with Test failures fix.

          Show
          Vinayakumar B added a comment - Attaching the patch with Test failures fix.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12636865/HDFS-6147.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12636865/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//console This message is automatically generated.
          Hide
          Vinayakumar B added a comment -

          Hi,
          Can some one take a look at the patch?

          Thanks,
          Vinay

          Show
          Vinayakumar B added a comment - Hi, Can some one take a look at the patch? Thanks, Vinay
          Hide
          Akira AJISAKA added a comment -
          +  public static final String  DFS_DATANODE_SCAN_USE_RANDOM_INITIAL_KEY = "dfs.datanode.scan.use.random.initial";;
          

          There are two semicolons. A semicolon can be removed.
          Other than that the patch looks good to me.

          Show
          Akira AJISAKA added a comment - + public static final String DFS_DATANODE_SCAN_USE_RANDOM_INITIAL_KEY = "dfs.datanode.scan.use.random.initial" ;; There are two semicolons. A semicolon can be removed. Other than that the patch looks good to me.
          Hide
          Vinayakumar B added a comment -

          Addressed above comments

          Show
          Vinayakumar B added a comment - Addressed above comments
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641447/HDFS-6147.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641447/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//console This message is automatically generated.
          Hide
          Akira AJISAKA added a comment -

          Thanks Vinayakumar B for the update. +1 (non-binding).
          In addition, Jenkins said +1 for the patch. https://builds.apache.org/job/PreCommit-HDFS-Build/6700/

          Show
          Akira AJISAKA added a comment - Thanks Vinayakumar B for the update. +1 (non-binding). In addition, Jenkins said +1 for the patch. https://builds.apache.org/job/PreCommit-HDFS-Build/6700/
          Hide
          Vinayakumar B added a comment -

          Rebased the patch on latest trunk.

          Show
          Vinayakumar B added a comment - Rebased the patch on latest trunk.
          Hide
          Vinayakumar B added a comment -

          rebased and Updated test.
          Please review

          Show
          Vinayakumar B added a comment - rebased and Updated test. Please review
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657553/HDFS-6147.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657553/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//console This message is automatically generated.
          Hide
          Vinayakumar B added a comment -

          Rebased the patch.
          Hi Uma Maheswara Rao G, can you take a look at the patch?

          Show
          Vinayakumar B added a comment - Rebased the patch. Hi Uma Maheswara Rao G , can you take a look at the patch?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666666/HDFS-6147.patch
          against trunk revision 6104520.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancTests
          org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666666/HDFS-6147.patch against trunk revision 6104520. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.TestDatanodeBlockScanner The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancTests org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//console This message is automatically generated.
          Hide
          Vinayakumar B added a comment -

          Since the block scanner is re-written, this issue no longer holds.

          Show
          Vinayakumar B added a comment - Since the block scanner is re-written, this issue no longer holds.

            People

            • Assignee:
              Vinayakumar B
              Reporter:
              Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development