Jobs failed during job initalization are never removed from Capacity Schedulers waiting list

• Key: HADOOP-5719
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Sreekanth Ramakrishnan
• Reporter: Sreekanth Ramakrishnan
• Fix Version/s: 0.20.1

Description

Jobs which fail during initalization are never removed from Capacity Schedulers waiting job list.

Comments

[Sreekanth Ramakrishnan]

Currently job initialization is done in JobInitalizationPoller in the poller when an Exception is thrown while doing JobInProgress.initTasks() it does a JobInProgress.fail() but the fail does not inform all the job in progress listeners, resulting in job not being removed from the waiting job list of scheduler.

[Sreekanth Ramakrishnan]

Attaching a fix which addresses this issue. Alongwith test case which tests the condition.

[Sreekanth Ramakrishnan]

The result of ant test-patch is :

     [exec] 
     [exec] 
     [exec] 
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

[Vinod K V]

Few comments:

• I think a better place for job removal from the JobQueuesManager is the cleanUpInitializedJobsList() method of teh JobInitializationPoller. We may want to rename this method and change its javadoc a bit.
• We don't need the null check before job.fail() in the initialization-poller
• Test-case doesn't compile(perhaps because of HADOOP-5726). Need to change the signature of FakeQueueInfo
• In the test-case, the statements depicting the error scenarios have to be reversed. For e.g. at TestCapacityScheduler.java +2045

  assertFalse("Waiting job does not contain submitted job",  mgr.getWaitingJobs("default").contains(job));

  should instead be

  assertFalse("Waiting job contains submitted job", mgr.getWaitingJobs("default").contains(job));

[Sreekanth Ramakrishnan]

Attaching patch incorporating most of Vinod's comments.

I think a better place for job removal from the JobQueuesManager is the cleanUpInitializedJobsList() method of teh JobInitializationPoller. We may want to rename this method and change its javadoc a bit.

This has not been incorporated because of issue described in HADOOP-5020 it is hit when JobInProgress.initTasks() throws an exception and terminate job is called and Capacity scheduler would never be able to remove the job from the waiting queue.

Also added a new test case TestJobInitalizationPoller which uses MiniMRCluster to verify if jobs failing initialization are actually removed from waiting queue.

[Sreekanth Ramakrishnan]

Removing an unused import statement from JobInitializationPoller

[Sreekanth Ramakrishnan]

Attaching patch incorporating Vinod's offline comments:

• Adding apache license to new test case file.
• Changed the name of new test case from TestJobInitializationPoller to TestJobInitialization.
• Corrected Typo of initialization the TestCapacityScheduler.
• Removed unnecessary whitespaces.
• Removed unused variable from TestJobInitialization
• Removed unused import from TestCapacityScheduler

[Vinod K V]

+1 for the patch.

[Sreekanth Ramakrishnan]

Output from ant test-patch:

     [exec]
     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]

[Sreekanth Ramakrishnan]

Attaching patch for branch 20.

[Sreekanth Ramakrishnan]

Attaching patch incorporating Hemanth's offline comments.

In TestJobInitialization

• Changed the scheduler property from guaranteed-capacity to capacity.
• Asserting if the submitted job is failed.

Attaching both 20 branch and trunk patch

[Hemanth Yamijala]

Changes look fine to me. +1.

[Sreekanth Ramakrishnan]

Output for ant test-patch for the latest attachment is as follows:

     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20. Thanks, Sreekanth !

[Hudson]

Integrated in Hadoop-trunk #828 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/828/)
. Forgot to add a new file in the previous commit.
. Remove jobs that failed initialization from the waiting queue in the capacity scheduler. Contributed by Sreekanth Ramakrishnan.