Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0.0-alpha
-
Fix Version/s: 0.23.3, 2.0.2-alpha
-
Component/s: resourcemanager
-
Labels:None
-
Target Version/s:
Description
It doesn't happen on my desktop, but it happens frequently during the builds with clover enabled. Surefire will report it as fork failure.
Activity
The test was calling ReourceManager#stop() after it thought it's done. This hit GenericEventHandler with an interrupt while it was trying to enqueue an event. This bubbled up and hit the RM's EventProcessor loop, which did System.exit(-1). It checks whether the JVM is being shutdown, but this is before ShutdownHookManager is activated.
It seems EventProcessor shouldn't do exit(-1) if it's got an exception during shutdown.
EventProcessor#stop() is setting
{stopped}to true before interrupting the thread. So we could just add one more condition and let it terminate normally.
while (!stopped && !Thread.currentThread().isInterrupted()) {
try {
event = eventQueue.take();
} catch (InterruptedException e) {
LOG.error("Returning, interrupted : " + e);
return; // TODO: Kill RM.
}
try {
scheduler.handle(event);
} catch (Throwable t) {
+ if (stopped) {
+ LOG.warn("Exception during shutdown: ", t);
+ break;
+ }
LOG.fatal("Error in handling event type " + event.getType()
+ " to the scheduler", t);
if (shouldExitOnError
&& !ShutdownHookManager.get().isShutdownInProgress()) {
LOG.info("Exiting, bbye..");
System.exit(-1);
}
}
}
I tested this idea and it worked. With -Pclover, TestRM always fails. With the added check, it succeeds.
BEFORE
------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.yarn.server.resourcemanager.TestRM Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] hadoop-yarn-server-resourcemanager ................ FAILURE [5.180s] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException; nested exception is java.util.concurrent.ExecutionException: org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log -> [Help 1]
AFTER
------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.yarn.server.resourcemanager.TestRM Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.973 sec Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534044/mapreduce-4387.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2535//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2535//console
This message is automatically generated.
No additional test is included since it is fixing the bug that failed the test. The test passes after the patch.
+1 lgtm. Thanks Kihwal!
I committed this to trunk, branch-2, and branch-0.23.
Integrated in Hadoop-Hdfs-trunk #1095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1095/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Mapreduce-trunk-Commit #2446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2446/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Hdfs-trunk-Commit #2496 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2496/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Common-trunk-Commit #2428 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2428/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Mapreduce-trunk #1128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1128/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Hdfs-0.23-Build #305 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/305/)
merge -r 1358202:1358203 from branch-2. FIXES: MAPREDUCE-4387 (Revision 1358205)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358205
Files :
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM (Kihwal Lee via tgraves) (Revision 1358201)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358201
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
Here is the output from the test. Pardon the format. It was captured from directly running what surefire did.