Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3412

Refactor the scheduler out of the JobTracker

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added the ability to chose between many schedulers, and to limit the number of running tasks per job.

      Description

      First I would like warn you that my proposition is assumed to be very naive. I just hope that reading it won't make you lose time.

      The aim

      It seems to me that improving Hadoop scheduling could be very profitable. But, it is hard to implement and compare schedulers, because the scheduling logic is mixed within the rest of the JobTracker.
      This bug is the first step of an attempt to improve the Hadoop scheduler. It re-implements the current scheduling algorithm in a separate class called JobScheduler. This new class is instantiated in the JobTracker.

      Bug fixed as a side effects

      This patch probably cannot be submited as it is.
      A first difficulty is that it does not have exactly the same behaviour than the current JobTracker. More precisely, it doesn't re-implement things like code that seems to be never called or concurency problems.
      I wrote TOCONFIRM where my proposition differ from the current implementation, so you can find them easily.
      I know that fixing bugs silently is bad. So, independently of what you decide about this patch, I will open issues for bugs that you confirm.

      Other side effects

      Another side effect of this patch is to add documentation about each step of the scheduling. I hope that it will help future improvement by lowering the level required to contribute to the scheduler.
      It also reduces the complexity and the granularity of the JobTracker (making it more parallel).

      The future

      If you feel that this is a step the right direction, I will try to propose a JobSchedulerInterface that many JobSchedulers could implement and to propose alternatives to the current « FifoJobScheduler ». If some of you have ideas about that please tell ^^ I will also open issues for things marked as FIXME in the patch.

        Attachments

        1. JobScheduler-v12.1.patch
          67 kB
          Tom White
        2. JobScheduler-v12.patch
          67 kB
          Tom White
        3. JobScheduler-v11.patch
          60 kB
          Tom White
        4. JobScheduler-v10.patch
          46 kB
          Tom White
        5. JobScheduler-v9.2.patch
          29 kB
          Matei Zaharia
        6. JobScheduler-v9.1.patch
          37 kB
          Tom White
        7. JobScheduler-v9.patch
          37 kB
          Tom White
        8. JobScheduler_v8.patch
          57 kB
          Brice Arnould
        9. JobScheduler_v7.1.patch
          52 kB
          Brice Arnould
        10. JobScheduler_v7.patch
          50 kB
          Brice Arnould
        11. JobScheduler_v6.4.patch
          49 kB
          Brice Arnould
        12. JobScheduler_v6.3.patch
          50 kB
          Brice Arnould
        13. SimpleResourceAwareJobScheduler.java
          8 kB
          Brice Arnould
        14. JobScheduler_v6.2.patch
          46 kB
          Brice Arnould
        15. JobScheduler_v6.1.patch
          41 kB
          Brice Arnould
        16. JobScheduler_v6.patch
          40 kB
          Brice Arnould
        17. JobScheduler_v5.patch
          40 kB
          Brice Arnould
        18. RackAwareJobScheduler.java
          2 kB
          Brice Arnould
        19. JobScheduler_v4.patch
          37 kB
          Brice Arnould
        20. JobScheduler_v3b.patch
          39 kB
          Brice Arnould
        21. JobScheduler_v3.patch
          39 kB
          Brice Arnould
        22. JobScheduler_v2.patch
          34 kB
          Brice Arnould
        23. JobScheduler.patch
          30 kB
          Brice Arnould

          Issue Links

            Activity

              People

              • Assignee:
                un_brice Brice Arnould
                Reporter:
                un_brice Brice Arnould
              • Votes:
                1 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: