Remove pre-emption from the capacity scheduler code base

• Key: HADOOP-5726
• Type: Improvement
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: rahul k singh
• Reporter: Hemanth Yamijala
• Fix Version/s: 0.20.1

Description

In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?

Comments

[rahul k singh]

Attaching the patch .

[Arun C Murthy]

+1 for the direction.

I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

Thoughts?

[rahul k singh]

[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 7 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================

[Hemanth Yamijala]

I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo.

Can you please open a new jira with your ideas and we can discuss there ?

[Hemanth Yamijala]

At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon.

On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20. Thanks, Rahul !

[Hudson]

Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/)
. Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.

[Vinod K V]

Just for the sake of record: If ever, we wish to bring back reclamation, we should reopen and fix MAPREDUCE-515.