Hadoop Common
  1. Hadoop Common
  2. HADOOP-4035

Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.

      Description

      HADOOP-3759 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc.

      1. 4035.1.patch
        31 kB
        Vinod Kumar Vavilapalli
      2. HADOOP-4035-20080918.1.txt
        56 kB
        Vinod Kumar Vavilapalli
      3. HADOOP-4035-20081006.txt
        75 kB
        Vinod Kumar Vavilapalli
      4. HADOOP-4035-20081006.1.txt
        89 kB
        Vinod Kumar Vavilapalli
      5. HADOOP-4035-20081008.txt
        98 kB
        Vinod Kumar Vavilapalli
      6. HADOOP-4035-20081121.txt
        83 kB
        Vinod Kumar Vavilapalli
      7. HADOOP-4035-20081126.1.txt
        101 kB
        Vinod Kumar Vavilapalli
      8. HADOOP-4035-20081128-4.txt
        128 kB
        Vinod Kumar Vavilapalli
      9. HADOOP-4035-20081202.txt
        135 kB
        Vinod Kumar Vavilapalli
      10. HADOOP-4035-20081202.1.txt
        135 kB
        Hemanth Yamijala
      11. HADOOP-4035-20081202.2.txt
        135 kB
        Hemanth Yamijala

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is depended upon by HADOOP-4523 [ HADOOP-4523 ]
          Gavin made changes -
          Link This issue blocks HADOOP-4523 [ HADOOP-4523 ]
          Owen O'Malley made changes -
          Component/s contrib/capacity-sched [ 12312466 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Release Note Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Hemanth Yamijala made changes -
          Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hemanth Yamijala made changes -
          Attachment HADOOP-4035-20081202.2.txt [ 12395083 ]
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hemanth Yamijala made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Hemanth Yamijala made changes -
          Attachment HADOOP-4035-20081202.1.txt [ 12395076 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081202.txt [ 12395067 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081128-4.txt [ 12394906 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081126.1.txt [ 12394718 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081121.txt [ 12394421 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Owen O'Malley made changes -
          Link This issue blocks HADOOP-4523 [ HADOOP-4523 ]
          Owen O'Malley made changes -
          Fix Version/s 0.19.0 [ 12313211 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by HADOOP-4236 [ HADOOP-4236 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by HADOOP-4236 [ HADOOP-4236 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081008.txt [ 12391762 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by HADOOP-4287 [ HADOOP-4287 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by HADOOP-4236 [ HADOOP-4236 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081006.1.txt [ 12391539 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20081006.txt [ 12391530 ]
          Hemanth Yamijala made changes -
          Component/s contrib/capacity-sched [ 12312466 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment HADOOP-4035-20080918.1.txt [ 12390384 ]
          Vinod Kumar Vavilapalli made changes -
          Assignee Vivek Ratan [ vivekr ] Vinod K V [ vinodkv ]
          Vinod Kumar Vavilapalli made changes -
          Attachment 4035.1.patch [ 12390241 ]
          Sameer Paranjpye made changes -
          Link This issue is part of HADOOP-3444 [ HADOOP-3444 ]
          Runping Qi made changes -
          Link This issue is related to HADOOP-2776 [ HADOOP-2776 ]
          Hemanth Yamijala made changes -
          Field Original Value New Value
          Description HADOOP-3749 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc. HADOOP-3759 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc.
          Hemanth Yamijala created issue -

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development