Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In refreshVolumes, we are creating executor service and submitting volume addition tasks to it.
      But we are not shutting down the service after the use. Even though we are not holding instance level service, the initialized thread could be left out.

      ExecutorService service = Executors.newFixedThreadPool(
                  changedVolumes.newLocations.size());
      

      So, simple fix for this would be to shutdown the service after its use.

      1. HDFS-9141.00.patch
        3 kB
        Uma Maheswara Rao G

        Activity

        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Updated the patch to handle this.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Updated the patch to handle this.
        Hide
        hitliuyi Yi Liu added a comment -

        +1 pending Jenkins, Thanks Uma.

        Show
        hitliuyi Yi Liu added a comment - +1 pending Jenkins, Thanks Uma.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 48s There were no new javac warning messages.
        +1 javadoc 11m 29s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 33s The applied patch generated 1 new checkstyle issues (total was 145, now 143).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 40s Pre-build of native portion
        -1 hdfs tests 127m 41s Tests failed in hadoop-hdfs.
            179m 19s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestEditLog
          org.apache.hadoop.hdfs.security.token.block.TestBlockToken



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12764189/HDFS-9141.00.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 151fca5
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12736/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12736/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 48s There were no new javac warning messages. +1 javadoc 11m 29s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 33s The applied patch generated 1 new checkstyle issues (total was 145, now 143). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 40s Pre-build of native portion -1 hdfs tests 127m 41s Tests failed in hadoop-hdfs.     179m 19s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.ha.TestEditLogTailer Timed out tests org.apache.hadoop.hdfs.server.namenode.TestEditLog   org.apache.hadoop.hdfs.security.token.block.TestBlockToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764189/HDFS-9141.00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 151fca5 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12736/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12736/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12736/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        The test failures are unrelated, and checkstyle is about the file contains too much lines, and we don't need to fix it.
        Will commit the patch shortly.

        Show
        hitliuyi Yi Liu added a comment - The test failures are unrelated, and checkstyle is about the file contains too much lines, and we don't need to fix it. Will commit the patch shortly.
        Hide
        hitliuyi Yi Liu added a comment -

        Committed to trunk and branch-2, thanks Uma.

        Show
        hitliuyi Yi Liu added a comment - Committed to trunk and branch-2, thanks Uma.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8539 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8539/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8539 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8539/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #466 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/466/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #466 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/466/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1197/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1197/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #459 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/459/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #459 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/459/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2402 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2402/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2402 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2402/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2374/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2374/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #434 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/434/)
        HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #434 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/434/ ) HDFS-9141 . Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G via yliu) (yliu: rev 715dbddf77866bb47a4b95421091f64a3785444f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java

          People

          • Assignee:
            umamaheswararao Uma Maheswara Rao G
            Reporter:
            umamaheswararao Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development