Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6147

New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      New blocks scanning will be delayed if old blocks deleted after datanode restart.

      Steps:
      1. Write some blocks and wait till all scans over
      2. Restart the datanode
      3. Delete some of the blocks
      4. Write new blocks which are less in size compared to deleted blocks.

      Problem:
      BlockPoolSliceScanner#updateBytesToScan(..) updates bytesLeft based on following comparison

         if (lastScanTime < currentPeriodStart) {
            bytesLeft += len;
          }

      But in BlockPoolSliceScanner#assignInitialVerificationTimes() bytesLeft decremented using below comparison

      if (now - entry.verificationTime < scanPeriod) {

      Hence when the old blocks are deleted bytesLeft going negative.
      new blocks will not be scanned until it becomes positive again.

      So in both places verificationtime should be compared against scanperiod.

      1. HDFS-6147.patch
        14 kB
        Vinayakumar B
      2. HDFS-6147.patch
        14 kB
        Vinayakumar B
      3. HDFS-6147.patch
        13 kB
        Vinayakumar B
      4. HDFS-6147.patch
        13 kB
        Vinayakumar B
      5. HDFS-6147.patch
        13 kB
        Vinayakumar B
      6. HDFS-6147.patch
        8 kB
        Vinayakumar B

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12636312/HDFS-6147.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
        org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
        org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
        org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
        org.apache.hadoop.hdfs.TestDatanodeBlockScanner

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12636312/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6472//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Attaching the patch with Test failures fix.

        Show
        Vinayakumar B added a comment - Attaching the patch with Test failures fix.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12636865/HDFS-6147.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12636865/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6510//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Hi,
        Can some one take a look at the patch?

        Thanks,
        Vinay

        Show
        Vinayakumar B added a comment - Hi, Can some one take a look at the patch? Thanks, Vinay
        Hide
        Akira AJISAKA added a comment -
        +  public static final String  DFS_DATANODE_SCAN_USE_RANDOM_INITIAL_KEY = "dfs.datanode.scan.use.random.initial";;
        

        There are two semicolons. A semicolon can be removed.
        Other than that the patch looks good to me.

        Show
        Akira AJISAKA added a comment - + public static final String DFS_DATANODE_SCAN_USE_RANDOM_INITIAL_KEY = "dfs.datanode.scan.use.random.initial" ;; There are two semicolons. A semicolon can be removed. Other than that the patch looks good to me.
        Hide
        Vinayakumar B added a comment -

        Addressed above comments

        Show
        Vinayakumar B added a comment - Addressed above comments
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12641447/HDFS-6147.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641447/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6700//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        Thanks Vinayakumar B for the update. +1 (non-binding).
        In addition, Jenkins said +1 for the patch. https://builds.apache.org/job/PreCommit-HDFS-Build/6700/

        Show
        Akira AJISAKA added a comment - Thanks Vinayakumar B for the update. +1 (non-binding). In addition, Jenkins said +1 for the patch. https://builds.apache.org/job/PreCommit-HDFS-Build/6700/
        Hide
        Vinayakumar B added a comment -

        Rebased the patch on latest trunk.

        Show
        Vinayakumar B added a comment - Rebased the patch on latest trunk.
        Hide
        Vinayakumar B added a comment -

        rebased and Updated test.
        Please review

        Show
        Vinayakumar B added a comment - rebased and Updated test. Please review
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12657553/HDFS-6147.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657553/HDFS-6147.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7455//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Rebased the patch.
        Hi Uma Maheswara Rao G, can you take a look at the patch?

        Show
        Vinayakumar B added a comment - Rebased the patch. Hi Uma Maheswara Rao G , can you take a look at the patch?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666666/HDFS-6147.patch
        against trunk revision 6104520.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
        org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
        org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
        org.apache.hadoop.hdfs.TestDatanodeBlockScanner

        The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.balancer.TestBalancTests
        org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666666/HDFS-6147.patch against trunk revision 6104520. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.TestDatanodeBlockScanner The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancTests org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7910//console This message is automatically generated.

          People

          • Assignee:
            Vinayakumar B
            Reporter:
            Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development