Hadoop Common
  1. Hadoop Common
  2. HADOOP-5726

Remove pre-emption from the capacity scheduler code base

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.
      Show
      Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.

      Description

      In an effort to simplify the code base, we would like to remove the pre-emption related code in the capacity scheduler. We would reintroduce this (possibly with some revisions to the original design) after a while. This will be an incompatible change. Any objections ?

        Issue Links

          Activity

          Hemanth Yamijala created issue -
          Hide
          rahul k singh added a comment -

          Attaching the patch .

          Show
          rahul k singh added a comment - Attaching the patch .
          rahul k singh made changes -
          Field Original Value New Value
          Attachment cs-without-preemption-23-1-2009.patch [ 12406210 ]
          Sreekanth Ramakrishnan made changes -
          Link This issue blocks HADOOP-5736 [ HADOOP-5736 ]
          Hide
          Arun C Murthy added a comment -

          +1 for the direction.

          I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

          Thoughts?

          Show
          Arun C Murthy added a comment - +1 for the direction. I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira? Thoughts?
          Hide
          rahul k singh added a comment -

          [exec]
          [exec]
          [exec]
          [exec]
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 7 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================

          Show
          rahul k singh added a comment - [exec] [exec] [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 7 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
          rahul k singh made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hemanth Yamijala added a comment -

          I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira?

          Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo.

          Can you please open a new jira with your ideas and we can discuss there ?

          Show
          Hemanth Yamijala added a comment - I'd propose we take this opportunity to further improve the capacity-scheduler by simplifying it's design, refactoring inner-classes into standalone classes, using the JobChangeEvents more extensively rather than re-sort queues on every heartbeat etc. Maybe a new jira? Arun, I agree. Particularly with the refactoring of inner classes into standalone classes. Also, I think we should look at combining all queue state into a single class and promote it to a first class object. Today, this state is split between JobQueueManager and QueueSchedulingInfo. Can you please open a new jira with your ideas and we can discuss there ?
          Hide
          Hemanth Yamijala added a comment -

          At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon.

          On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?

          Show
          Hemanth Yamijala added a comment - At this point, it looks like there's no objection to the removal of this code. I plan to commit this soon. On a related note, though this is a major change, it appears that committing this to the 0.20 branch will significantly simplify further development. Is there an objection to this ?
          Hide
          Hemanth Yamijala added a comment -

          I just committed this to trunk and branch 0.20. Thanks, Rahul !

          Show
          Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.20. Thanks, Rahul !
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Release Note Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees.
          Fix Version/s 0.20.1 [ 12313866 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/)
          . Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/ ) . Remove pre-emption from capacity scheduler code base. Contributed by Rahul Kumar Singh.
          Owen O'Malley made changes -
          Component/s contrib/capacity-sched [ 12312466 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Just for the sake of record: If ever, we wish to bring back reclamation, we should reopen and fix MAPREDUCE-515.

          Show
          Vinod Kumar Vavilapalli added a comment - Just for the sake of record: If ever, we wish to bring back reclamation, we should reopen and fix MAPREDUCE-515 .
          Hide
          Ryan Hennig added a comment -

          Are there any plans to re-add pre-emption to the Capacity Scheduler? Or any best practices for using the CS in a way that allows high-priority, time-sensitive jobs to have adequate resources when needed, without sacrificing utilization at other times?

          Show
          Ryan Hennig added a comment - Are there any plans to re-add pre-emption to the Capacity Scheduler? Or any best practices for using the CS in a way that allows high-priority, time-sensitive jobs to have adequate resources when needed, without sacrificing utilization at other times?

            People

            • Assignee:
              rahul k singh
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development