Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4372

Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor and Shutdown hook manager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2, resourcemanager
    • Labels:
      None

      Description

      Please find the attached resource manager thread dump for the issue.

      1. rm-threaddump.out
        8 kB
        Devaraj K
      2. MAPREDUCE-4372.patch
        1 kB
        Devaraj K
      3. MAPREDUCE-4372-1.patch
        2 kB
        Devaraj K

        Activity

        Devaraj K created issue -
        Devaraj K made changes -
        Field Original Value New Value
        Attachment rm-threaddump.out [ 12533463 ]
        Hide
        Devaraj K added a comment -

        It seems very similar to MAPREDUCE-4031.

        Show
        Devaraj K added a comment - It seems very similar to MAPREDUCE-4031 .
        Devaraj K made changes -
        Attachment MAPREDUCE-4372.patch [ 12533468 ]
        Hide
        Devaraj K added a comment -

        Attaching straightforward patch to fix the issue.

        Show
        Devaraj K added a comment - Attaching straightforward patch to fix the issue.
        Devaraj K made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12533468/MAPREDUCE-4372.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2514//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2514//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533468/MAPREDUCE-4372.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2514//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2514//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good, I have one minor comment.

        I would prefer to see

        (ShutdownHookManager.get().isShutdownInProgress()) == false
        

        as

        !ShutdownHookManager.get().isShutdownInProgress()
        

        The javadoc warnings appear to be cause by MAPREDUCE-4355

        [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java:651: warning - @param argument "jobid" is not a parameter name.
        [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java:669: warning - @param argument "jobid" is not a parameter name.
        
        Show
        Robert Joseph Evans added a comment - The patch looks good, I have one minor comment. I would prefer to see (ShutdownHookManager.get().isShutdownInProgress()) == false as !ShutdownHookManager.get().isShutdownInProgress() The javadoc warnings appear to be cause by MAPREDUCE-4355 [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java:651: warning - @param argument "jobid" is not a parameter name. [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java:669: warning - @param argument "jobid" is not a parameter name.
        Devaraj K made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Devaraj K made changes -
        Attachment MAPREDUCE-4372-1.patch [ 12533636 ]
        Hide
        Devaraj K added a comment -

        Thanks a lot Robert for looking into the patch. I have updated the patch as per your suggestion.

        Show
        Devaraj K added a comment - Thanks a lot Robert for looking into the patch. I have updated the patch as per your suggestion.
        Devaraj K made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12533636/MAPREDUCE-4372-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2520//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2520//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533636/MAPREDUCE-4372-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2520//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2520//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Changes look good to me +1.

        Show
        Robert Joseph Evans added a comment - Changes look good to me +1.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Devaraj,

        I put this into trunk and branch-2.

        Show
        Robert Joseph Evans added a comment - Thanks Devaraj, I put this into trunk and branch-2.
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 2.0.1-alpha [ 12321442 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2462 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2462/)
        MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2462 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2462/ ) MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2393 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2393/)
        MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2393 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2393/ ) MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2412 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2412/)
        MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2412 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2412/ ) MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1090 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1090/)
        MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1090 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1090/ ) MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1123/)
        MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1123/ ) MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1354531) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354531 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Robert Joseph Evans made changes -
        Fix Version/s 0.23.3 [ 12320060 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #306 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/306/)
        svn merge -c 1354531 FIXES: MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1358402)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358402
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #306 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/306/ ) svn merge -c 1354531 FIXES: MAPREDUCE-4372 . Deadlock in Resource Manager (Devaraj K via bobby) (Revision 1358402) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358402 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Fix Version/s 2.1.0-alpha [ 12321442 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        22h 1m 1 Devaraj K 27/Jun/12 11:10
        Open Open Patch Available Patch Available
        1h 9m 2 Devaraj K 27/Jun/12 11:11
        Patch Available Patch Available Resolved Resolved
        4h 18m 1 Robert Joseph Evans 27/Jun/12 15:29
        Resolved Resolved Closed Closed
        106d 3h 19m 1 Arun C Murthy 11/Oct/12 18:48

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development