Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4139

Potential ResourceManager deadlock when SchedulerEventDispatcher is stopped

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      When the main thread calls ResourceManager$SchedulerEventDispatcher.stop() it grabs a lock on the object, kicks the event processor thread, and then waits for the thread to exit. However the interrupted event processor thread can end up trying to call the synchronized getConfig() method which results in deadlock.

      1. MAPREDUCE-4139.patch
        6 kB
        Jason Lowe
      2. MAPREDUCE-4139.patch
        6 kB
        Jason Lowe

        Issue Links

          Activity

          Jason Lowe created issue -
          Hide
          Jason Lowe added a comment -

          Patch to cache config's exit on error value upon startup so we can avoid the getConfig() call when fielding errors in the handling thread.

          Also changed the handling thread so an InterruptedException from a handler is treated the same as an interrupted wait on the queue. Since the handler thread interrupt is asynchronous, it seemed odd to treat it differently based on whether we were within a handler method.

          Show
          Jason Lowe added a comment - Patch to cache config's exit on error value upon startup so we can avoid the getConfig() call when fielding errors in the handling thread. Also changed the handling thread so an InterruptedException from a handler is treated the same as an interrupted wait on the queue. Since the handler thread interrupt is asynchronous, it seemed odd to treat it differently based on whether we were within a handler method.
          Jason Lowe made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-4139.patch [ 12522510 ]
          Jason Lowe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3 [ 12320060 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522510/MAPREDUCE-4139.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
          org.apache.hadoop.yarn.server.TestDiskFailures
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.ipc.TestSocketFactory
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522510/MAPREDUCE-4139.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.server.TestDiskFailures org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//console This message is automatically generated.
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jason Lowe added a comment -

          Updated patch to fix for findbug warning.

          Test failures are unrelated to this patch. They are the same failures that were recently reported for MAPREDUCE-4144 and others.

          Show
          Jason Lowe added a comment - Updated patch to fix for findbug warning. Test failures are unrelated to this patch. They are the same failures that were recently reported for MAPREDUCE-4144 and others.
          Jason Lowe made changes -
          Attachment MAPREDUCE-4139.patch [ 12522569 ]
          Jason Lowe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522569/MAPREDUCE-4139.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522569/MAPREDUCE-4139.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//console This message is automatically generated.
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jason Lowe made changes -
          Link This issue is part of MAPREDUCE-3942 [ MAPREDUCE-3942 ]
          Hide
          Jason Lowe added a comment -

          This ended up being fixed by MAPREDUCE-3942.

          Show
          Jason Lowe added a comment - This ended up being fixed by MAPREDUCE-3942 .
          Jason Lowe made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322471 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1d 3h 41m 2 Jason Lowe 13/Apr/12 13:52
          Patch Available Patch Available Open Open
          4d 21h 24m 2 Jason Lowe 17/Apr/12 22:35
          Open Open Resolved Resolved
          6m 21s 1 Jason Lowe 17/Apr/12 22:41
          Resolved Resolved Closed Closed
          176d 19h 6m 1 Arun C Murthy 11/Oct/12 17:48

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development