Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3684

LocalDistributedCacheManager does not shut down its thread pool

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This was observed by running a Hive job in local mode. The job completed but the client process did not exit for 60 seconds.

        Activity

        Hide
        Tom White added a comment -

        This patch ensures that the executor service is shut down in a finally block. I also added thread factories in LocalJobRunner and LocalDistributedCacheManager so that threads have meaningful names.

        There's no unit test, but I manually verified this change by running the same Hive query against it and observing that there was no 60 second pause.

        Show
        Tom White added a comment - This patch ensures that the executor service is shut down in a finally block. I also added thread factories in LocalJobRunner and LocalDistributedCacheManager so that threads have meaningful names. There's no unit test, but I manually verified this change by running the same Hive query against it and observing that there was no 60 second pause.
        Hide
        Carl Steinbach added a comment -

        I also verified that this patch resolves the problem.

        Show
        Carl Steinbach added a comment - I also verified that this patch resolves the problem.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12510915/MAPREDUCE-3684.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1623//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1623//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510915/MAPREDUCE-3684.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1623//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1623//console This message is automatically generated.
        Hide
        Ahmed Radwan added a comment -

        The patch looks good to me +1, thanks Tom! However I wasn't able to reproduce the issue, I tried a pi job in local mode with and without the patch and I haven't seen such delay in either case. I guess I am doing something wrong, should I see this delay with other jobs (not just hive)?

        Show
        Ahmed Radwan added a comment - The patch looks good to me +1, thanks Tom! However I wasn't able to reproduce the issue, I tried a pi job in local mode with and without the patch and I haven't seen such delay in either case. I guess I am doing something wrong, should I see this delay with other jobs (not just hive)?
        Hide
        Tom White added a comment -

        I couldn't reproduce it with a pi job either - I think Hive was triggering it since it uses the distributed cache, which in local mode is where the executor service runs and was not being properly shut down.

        Thanks for the review - I will commit this soon.

        Show
        Tom White added a comment - I couldn't reproduce it with a pi job either - I think Hive was triggering it since it uses the distributed cache, which in local mode is where the executor service runs and was not being properly shut down. Thanks for the review - I will commit this soon.
        Hide
        Ahmed Radwan added a comment -

        Thanks Tom! +1

        Show
        Ahmed Radwan added a comment - Thanks Tom! +1
        Hide
        Alejandro Abdelnur added a comment -

        +1

        Show
        Alejandro Abdelnur added a comment - +1
        Hide
        Tom White added a comment -

        I've just committed this.

        Show
        Tom White added a comment - I've just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #401 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/401/)
        Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #401 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/401/ ) Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1627/)
        MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread pool.

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1627 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1627/ ) MAPREDUCE-3684 . LocalDistributedCacheManager does not shut down its thread pool. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1572 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1572/)
        MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread pool.

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1572 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1572/ ) MAPREDUCE-3684 . LocalDistributedCacheManager does not shut down its thread pool. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #378 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/378/)
        Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #378 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/378/ ) Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1554 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1554/)
        MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread pool.

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1554 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1554/ ) MAPREDUCE-3684 . LocalDistributedCacheManager does not shut down its thread pool. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #388 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/388/)
        Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #388 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/388/ ) Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #930 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/930/)
        MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread pool.

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #930 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/930/ ) MAPREDUCE-3684 . LocalDistributedCacheManager does not shut down its thread pool. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #143 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/143/)
        Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #143 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/143/ ) Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #165 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/165/)
        Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #165 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/165/ ) Merge -r 1232980:1232981 from trunk to branch-0.23. Fixes: MAPREDUCE-3684 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #963 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/963/)
        MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread pool.

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #963 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/963/ ) MAPREDUCE-3684 . LocalDistributedCacheManager does not shut down its thread pool. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232981 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development