Amareshwari: The same thing happens in FairSchedulerServlet.java. Do you think we should file another JIRA or it can be fixed here in this one?

I think it can be fixed here. I see that they are related.

FairSchedulerServlet suffer from this problem seriously because it holds the JobTracker lock while looping through jobs.
So fixing this is important for FairScheduler.

I make both pages skip the uninitialized jobs.
Do you think this is a right fix?
I am still thinking how to test it. Any suggestion on the unit test?

The attached patch has problems applying. Maybe stale. Attaching a new patch (same code) what applies fine.

This looks good to me.
Writing a unit test would be hard here. But it can be manual testing, like holding a lock in debugger and triggering webUI refresh, would be very useful.
+1 modular manual testing.
Nit. Please add empty line before FairSchedulerServlet.getInitedJobs()

Thanks Priyo for rebase the patch.
Fix the empty line pointed out by Konstantin.

Hi Scott and Konstantin,
I tested by putting a breakpoint in JobInProgress.initTasks().

1) jobqueue_details.jsp is working fine i.e. not locking up
2) FairSchedulerServlet is locking up in showPools method as soon as it tries to synchronize on "scheduler". It continues again as soon as JIB.initTasks() finish. Not sure how this is happening as initTasks() lock JIB's monitor and showPools lock scheduler's monitor. Anyway, the patch doesn't seem to address the FSS issue.

> 2) FairSchedulerServlet is locking up in showPools method as soon as it tries to synchronize on "scheduler". It continues again as soon as JIB.initTasks() finish. Not sure how this is happening as initTasks() lock JIB's monitor and showPools lock scheduler's monitor. Anyway, the patch doesn't seem to address the FSS issue.

Thanks for the testing, Priyo.
Can you take a jstack at the moment that it waits for scheduler?
Then we can figure out the lock dependency.

Added 4 jstacks taken few seconds apart

This is what appears to be happening. JIB.initTasks() locks JIB. FairScheduler.assignTasks() has locked JT and FS and is waiting to lock JIB. Finally FairSchedulerServlet.showPools() is waiting to lock FS and is blocked.

Now the current case may have been aggravated because I had a breakpoint on initTasks() but on a busy cluster, assignTasks() must be happening extremely often so there is a very high probability that assignTasks() will block on initTasks() and thereby causing FSS to block as well.

Your thoughts?

Nice observation, Priyo. I think we can also fix the logic inside assignTasks() so it skips the uninitialized jobs.

Addressed the problem found by Priyo

Hi Scott,
I tested the new patch and this looks good. Looks like the patch applies fine on trunk but not on 0.22 because MAPREDUCE-1783 was added to trunk.

Hey Priyo,
Thanks again for the help. I will commit MAPREDUCE-1783 to 0.22.

test-patch result:

     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
     [exec]     +1 system test framework.  The patch passed system test framework compile.
     [exec]
     [exec]
     [exec]
     [exec]
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================

Patch was manually tested. Ant test passed. Looks good. +1

I just committed this. Thank you Scott and Priyo.

Integrated in Hadoop-Mapreduce-trunk-Commit #605 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/605/)
MAPREDUCE-1825. jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized. Contributed by Scott Chen.

Integrated in Hadoop-Mapreduce-trunk-Commit #606 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/606/)
MAPREDUCE-1825. A dot CHANGES.txt.

Thanks for the help, Konstantin and Priyo.

Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)