Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-516

Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed

    Description

      When a HighRAMJob turns up at the head of the queue, the current implementation of support for HighRAMJobs in the Capacity Scheduler has problem in that the scheduler stops assigning tasks to all TaskTrackers in the cluster until a HighRAMJob finds a suitable TaskTrackers for all its tasks.

      This causes a severe utilization problem since effectively no new tasks are allowed to run until the HighRAMJob (at the head of the queue) gets slots.

      Attachments

        1. mr516-ydist.patch
          170 kB
          Sreekanth Ramakrishnan
        2. MAPREDUCE-516-Y20.patch
          170 kB
          Sreekanth Ramakrishnan
        3. HADOOP-5964-12.patch
          168 kB
          Hemanth Yamijala
        4. HADOOP-5964-11.patch
          165 kB
          Hemanth Yamijala
        5. HADOOP-5964_9_20090619.patch
          164 kB
          Arun Murthy
        6. HADOOP-5964_9_20090619.patch
          163 kB
          Arun Murthy
        7. HADOOP-5964_8_20090618.patch
          163 kB
          Arun Murthy
        8. HADOOP-5964_7_20090618.patch
          148 kB
          Arun Murthy
        9. HADOOP-5964_6_20090617.patch
          145 kB
          Arun Murthy
        10. HADOOP-5964_4_20090615.patch
          84 kB
          Arun Murthy
        11. HADOOP-5964_2_20090629_yhadoop.patch
          170 kB
          Arun Murthy
        12. HADOOP-5964_2_20090629_yhadoop.patch
          170 kB
          Arun Murthy
        13. HADOOP-5964_2_20090609.patch
          68 kB
          Arun Murthy
        14. HADOOP-5964_11_20090623.patch
          174 kB
          Arun Murthy
        15. HADOOP-5964_10_20090622.patch
          162 kB
          Arun Murthy
        16. HADOOP-5964_1_20090623_yhadoop.patch
          170 kB
          Arun Murthy
        17. HADOOP-5964_1_20090608.patch
          54 kB
          Arun Murthy
        18. HADOOP-5964_0_20090602.patch
          31 kB
          Arun Murthy

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-5884 Capacity scheduler should account high memory jobs as using more capacity of the queue

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-5881 Simplify configuration related to task-memory-monitoring and memory-based scheduling

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-3789 CapacityTaskScheduler may perform unnecessary reservations in heterogenous tracker environments

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4001 Improve MAPREDUCE-3789's fix logic by looking at job's slot demands instead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              acmurthy Arun Murthy
              acmurthy Arun Murthy
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: