Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Hdfs-0.23-Build #305 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/305/)
merge -r 1358202:1358203 from branch-2. FIXES: MAPREDUCE-4387 (Revision 1358205)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358205
Files :

• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Mapreduce-trunk #1128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1128/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Common-trunk-Commit #2428 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2428/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Hdfs-trunk-Commit #2496 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2496/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Mapreduce-trunk-Commit #2446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2446/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

Integrated in Hadoop-Hdfs-trunk #1095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1095/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)

Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java

I committed this to trunk, branch-2, and branch-0.23.

+1 lgtm. Thanks Kihwal!

No additional test is included since it is fixing the bug that failed the test. The test passes after the patch.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534044/mapreduce-4387.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2535//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2535//console

This message is automatically generated.

I tested this idea and it worked. With -Pclover, TestRM always fails. With the added check, it succeeds.

BEFORE

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.yarn.server.resourcemanager.TestRM

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] hadoop-yarn-server-resourcemanager ................ FAILURE [5.180s]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException; nested exception is java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log -> [Help 1]

AFTER

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.yarn.server.resourcemanager.TestRM
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.973 sec

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

EventProcessor#stop() is setting

to true before interrupting the thread. So we could just add one more condition and let it terminate normally.

        while (!stopped && !Thread.currentThread().isInterrupted()) {
          try {
            event = eventQueue.take();
          } catch (InterruptedException e) {
            LOG.error("Returning, interrupted : " + e);
            return; // TODO: Kill RM.
          }

          try {
            scheduler.handle(event);
          } catch (Throwable t) {
+           if (stopped) {
+             LOG.warn("Exception during shutdown: ", t);
+             break;
+           }
            LOG.fatal("Error in handling event type " + event.getType()
                + " to the scheduler", t);
            if (shouldExitOnError
                && !ShutdownHookManager.get().isShutdownInProgress()) {
              LOG.info("Exiting, bbye..");
              System.exit(-1);
            }
          }
        }

The test was calling ReourceManager#stop() after it thought it's done. This hit GenericEventHandler with an interrupt while it was trying to enqueue an event. This bubbled up and hit the RM's EventProcessor loop, which did System.exit(-1). It checks whether the JVM is being shutdown, but this is before ShutdownHookManager is activated.

It seems EventProcessor shouldn't do exit(-1) if it's got an exception during shutdown.

Here is the output from the test. Pardon the format. It was captured from directly running what surefire did.

3,0001,2012-06-29 08:32:03\u002C950 FATAL [ResourceManager Event Processor]
resourcemanager.ResourceManager (ResourceManager.java:run(327)) - Error in
handling event type APP_REMOVED to the
scheduler\norg.apache.hadoop.yarn.YarnException:
java.lang.InterruptedException\n\tat
org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:184)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl$KillTransition.transition(RMContainerImpl.java:313)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl$KillTransition.transition(RMContainerImpl.java:303)\n\tat
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)\n\tat
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)\n\tat
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)\n\tat
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:200)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.containerCompleted(FifoScheduler.java:711)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.doneApplication(FifoScheduler.java:325)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:643)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:92)\n\tat
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:325)\n\tat
java.lang.Thread.run(Thread.java:662)\nCaused by:
java.lang.InterruptedException\n\tat
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)\n\tat
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)\n\tat
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)\n\tat
org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:181)\n\t...
15 more\n
3,0001,2012-06-29 08:32:03\u002C952 INFO  [ResourceManager Event Processor]
resourcemanager.ResourceManager (ResourceManager.java:run(331)) - Exiting\u002C
bbye..\n