I see this too. Here's a thread dump of the junit process while it appears to be waiting for something to finish.

At the very least the wait operation could do some timeouts so we'd get a better output in the log

Two issues are responsible for this. The first is HADOOP-5792, which caused TestQueueCapacities to fail. Once reverted, HADOOP-4981 evidently introduces a second regression, which was ignored because it was assumed that trunk remained the sole cause of the failure. Note that TestQueueCapacities is also failing in the 0.20 branch.

Currently, the TestQueueCapacities and MiniMRCluster tests in Capacity scheduler are broken because of Class not found exception, of Commons cli.

this patch adds commons-cli-1.2 to the capacity scheduler's classpath

The reason why the test case is currently timing out in the trunk is because, currently frame work returns setup and cleanup tasks always as a map task no matter which slot is free on task tracker. Causing setup and cleanup tasks to block. This causes the ControlledMapReduceJob to wait forever.

Attaching patch to fix this issue. Without this patch the test case does not pass, and with it it passes. Currently in TaskInProgress we check if the task is map task or reduce task and if the task is Setup or cleanup we assign appropriate split class and split bytes.

+1 for the patch. The issue was due to HADOOP-5850. Thanks Sreekanth for finding out.

Output from ant test-patch:

     [exec]
     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

The patch fixes a test case failure and the test actually tests the bug.

Same patch applies for branch 20.

Ant test passed on my local box.

My original comments were based on the results of a git-bisect from the last successful Hudson revision to HEAD (w/ 10 min timeout), identifying HADOOP-5792 as the first point where TestQueueCapacities started failing; dropping HADOOP-5792 from the tree identified HADOOP-4981 as a point where a failure resurfaced. I confirmed manually within the bisect branch before reopening the prenominate issues, but fresh subversion checkouts of the relevant revisions do not exhibit the behavior I reported previously. I must be mistaken. Thanks for doing the real work on this, Sreekanth.

After applying the patch to a fresh checkout, the test still times out on my machine...

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409841/HADOOP-5869-1.patch
against trunk revision 782083.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/474/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/474/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/474/console

This message is automatically generated.

Attaching new patch correcting few issues:

• The tests were failing randomly because of a timing issue with regards to speculative tasks being launched. The speculative execution is currently disabled in this patch.
• The tests were timing out instead of assertion failing because, in MiniMRCluster.shutdown() we do a waitTaskTrackers(). In case of controlled jobs the trackers never get idle until we finish tasks, but then assertion has failed and we would have wait for test to time out.

I had a chat with Sreekanth on this one. We think the change in MiniMRCluster to remove waitTaskTrackers is an improvement that is not required for this bug fix. At the same time it is a useful improvement to consider (though the fix is possibly wrong and needs change). We will move this change to a separate JIRA to discuss further on the MiniMRCluster changes.

Also, I don't see this patch on Hudson's queue, though its been a while since it was submitted. I would recommend that tests be run locally on the new patch to verify it is working and then I can commit it. The bug has been open for a way too long time.

Attaching patch incorporating Hemanths Comments.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12410207/HADOOP-5869-3.patch
against trunk revision 783059.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/486/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/486/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/486/console

This message is automatically generated.

The failure in HDFS proxy is not related to the changes made in this patch. The test case fails with IOException in Configuration.config, whereas patch does not change anything related to configuration.

Patch file for Hadoop 0.20. Only changes path to the test directory.

I committed this to trunk and the 0.20 branch. Thanks, Sreekanth !

TestQueueCapacities timed out again. (http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/868/).

HADOOP-6064 is the duplicate. I am resolving this issue because rather than fixing a different problem that shows up every time, I think it is better to just simplify this testcase drastically. That's the focus on HADOOP-6064. The other reason is that HADOOP-5869 is still a valid fix, and solves a problem correctly - just that now new symptoms are showing up. Please watch HADOOP-6064 if you are interested.