Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4035

Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.19.0
    • 0.20.0
    • None
    • None
    • Incompatible change, Reviewed
    • Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.

    Description

      HADOOP-3759 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc.

      Attachments

        1. 4035.1.patch
          31 kB
          Vinod Kumar Vavilapalli
        2. HADOOP-4035-20080918.1.txt
          56 kB
          Vinod Kumar Vavilapalli
        3. HADOOP-4035-20081006.1.txt
          89 kB
          Vinod Kumar Vavilapalli
        4. HADOOP-4035-20081006.txt
          75 kB
          Vinod Kumar Vavilapalli
        5. HADOOP-4035-20081008.txt
          98 kB
          Vinod Kumar Vavilapalli
        6. HADOOP-4035-20081121.txt
          83 kB
          Vinod Kumar Vavilapalli
        7. HADOOP-4035-20081126.1.txt
          101 kB
          Vinod Kumar Vavilapalli
        8. HADOOP-4035-20081128-4.txt
          128 kB
          Vinod Kumar Vavilapalli
        9. HADOOP-4035-20081202.1.txt
          135 kB
          Hemanth Yamijala
        10. HADOOP-4035-20081202.2.txt
          135 kB
          Hemanth Yamijala
        11. HADOOP-4035-20081202.txt
          135 kB
          Vinod Kumar Vavilapalli

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vinodkv Vinod Kumar Vavilapalli
            yhemanth Hemanth Yamijala
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment