Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4139

Potential ResourceManager deadlock when SchedulerEventDispatcher is stopped

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      When the main thread calls ResourceManager$SchedulerEventDispatcher.stop() it grabs a lock on the object, kicks the event processor thread, and then waits for the thread to exit. However the interrupted event processor thread can end up trying to call the synchronized getConfig() method which results in deadlock.

      1. MAPREDUCE-4139.patch
        6 kB
        Jason Lowe
      2. MAPREDUCE-4139.patch
        6 kB
        Jason Lowe

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          This ended up being fixed by MAPREDUCE-3942.

          Show
          Jason Lowe added a comment - This ended up being fixed by MAPREDUCE-3942 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522569/MAPREDUCE-4139.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522569/MAPREDUCE-4139.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2219//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Updated patch to fix for findbug warning.

          Test failures are unrelated to this patch. They are the same failures that were recently reported for MAPREDUCE-4144 and others.

          Show
          Jason Lowe added a comment - Updated patch to fix for findbug warning. Test failures are unrelated to this patch. They are the same failures that were recently reported for MAPREDUCE-4144 and others.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522510/MAPREDUCE-4139.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
          org.apache.hadoop.yarn.server.TestDiskFailures
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.ipc.TestSocketFactory
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522510/MAPREDUCE-4139.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.server.TestDiskFailures org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2217//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Patch to cache config's exit on error value upon startup so we can avoid the getConfig() call when fielding errors in the handling thread.

          Also changed the handling thread so an InterruptedException from a handler is treated the same as an interrupted wait on the queue. Since the handler thread interrupt is asynchronous, it seemed odd to treat it differently based on whether we were within a handler method.

          Show
          Jason Lowe added a comment - Patch to cache config's exit on error value upon startup so we can avoid the getConfig() call when fielding errors in the handling thread. Also changed the handling thread so an InterruptedException from a handler is treated the same as an interrupted wait on the queue. Since the handler thread interrupt is asynchronous, it seemed odd to treat it differently based on whether we were within a handler method.

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development