Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3718

Datanode won't shutdown because of runaway DataBlockScanner thread

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Datanode sometimes does not shutdown because the block pool scanner thread keeps running. It prints out "Starting a new period" every five seconds, even after shutdown() is called. Somehow the interrupt is missed.

      DataBlockScanner will also terminate if datanode.shouldRun is false, but in DataNode#shutdown, DataBlockScanner#shutdown() is invoked before it is being set to false.

      Is there any reason why datanode.shouldRun is set to false later?

        Activity

        Hide
        Suresh Srinivas added a comment -

        There is no specific reason for that order. We should set DataNode#shouldRun to false before other shutdown calls.

        Show
        Suresh Srinivas added a comment - There is no specific reason for that order. We should set DataNode#shouldRun to false before other shutdown calls.
        Hide
        Kihwal Lee added a comment -

        Moved up the line where datanode.shouldRun is set to false. I ran all HDFS tests and there wasn't any new test failure.

        Show
        Kihwal Lee added a comment - Moved up the line where datanode.shouldRun is set to false. I ran all HDFS tests and there wasn't any new test failure.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12537784/hdfs-3718.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDatanodeBlockScanner
        org.apache.hadoop.hdfs.TestPersistBlocks

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2899//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2899//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537784/hdfs-3718.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2899//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2899//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch

        Show
        Suresh Srinivas added a comment - +1 for the patch
        Hide
        Daryn Sharp added a comment -

        I've committed to trunk, branches for 2.x, and 23. Thanks Kihwal!

        Show
        Daryn Sharp added a comment - I've committed to trunk, branches for 2.x, and 23. Thanks Kihwal!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2601 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2601/)
        HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2601 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2601/ ) HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2577 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2577/)
        HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2577 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2577/ ) HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2642 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2642/)
        HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2642 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2642/ ) HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/344/)
        svn merge -c 1373090 FIXES: HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373098)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373098
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/344/ ) svn merge -c 1373090 FIXES: HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373098) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373098 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/)
        HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/ ) HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/)
        HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/ ) HDFS-3718 . Datanode won't shutdown because of runaway DataBlockScanner thread (Kihwal Lee via daryn) (Revision 1373090) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373090 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Yang Jiandan added a comment -

        Why interrupt is missing?
        I found every place to throw InterruptedException has called Thread.interrupt to set interrupt flag。

        Show
        Yang Jiandan added a comment - Why interrupt is missing? I found every place to throw InterruptedException has called Thread.interrupt to set interrupt flag。

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development