Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4360

Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.1
    • Fix Version/s: 0.22.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-4360-22-1.patch
      12 kB
      Mayank Bansal
    2. MAPREDUCE-4360-22.patch
      12 kB
      Mayank Bansal

      Activity

      Hide
      Mayank Bansal added a comment -

      Maximum capacity set on a container queue is not getting honored. For ex: If Queue A has Queue A1 and Queue A2 as child queues (leaf queues), and if max capacity is not set for the child queues and set only to the container queue (Queue A), then all the slots occupied by the jobs submitted to Queue A1 and Queue A2 should not exceed the maximum capacity configured for Queue A. However, it is not the case now. Tasks gets scheduled beyond Queue A's configured max capacity.

      Show
      Mayank Bansal added a comment - Maximum capacity set on a container queue is not getting honored. For ex: If Queue A has Queue A1 and Queue A2 as child queues (leaf queues), and if max capacity is not set for the child queues and set only to the container queue (Queue A), then all the slots occupied by the jobs submitted to Queue A1 and Queue A2 should not exceed the maximum capacity configured for Queue A. However, it is not the case now. Tasks gets scheduled beyond Queue A's configured max capacity.
      Hide
      Mayank Bansal added a comment -

      Attaching the initial patch for 22.

      Thanks,
      Mayank

      Show
      Mayank Bansal added a comment - Attaching the initial patch for 22. Thanks, Mayank
      Hide
      Konstantin Shvachko added a comment -

      You should work on usual formatting issues.

      So the fix as I understood is to build a hierarchy of queues by setting the parent member. Then find the parent that has maxCapacity set, and verify it is not exceeded. Sounds like the right approach.
      Do you need to update the current capacity for the parent, when you allocate new slots for a child queue?

      Show
      Konstantin Shvachko added a comment - You should work on usual formatting issues. So the fix as I understood is to build a hierarchy of queues by setting the parent member. Then find the parent that has maxCapacity set, and verify it is not exceeded. Sounds like the right approach. Do you need to update the current capacity for the parent, when you allocate new slots for a child queue?
      Hide
      Jason Lowe added a comment -

      This JIRA indicates that trunk is affected, but I believe this has already been addressed in trunk (and branch-2 and branch-0.23) by MAPREDUCE-3683.

      Show
      Jason Lowe added a comment - This JIRA indicates that trunk is affected, but I believe this has already been addressed in trunk (and branch-2 and branch-0.23) by MAPREDUCE-3683 .
      Hide
      Mayank Bansal added a comment -

      Jason,

      I did not realize that it is already fixed in trunk will update the JIRA. Thanks for pointing this out.

      Konst,

      Thats already been done in when tasks been assigned any queue.

      Thanks,
      Mayank

      Show
      Mayank Bansal added a comment - Jason, I did not realize that it is already fixed in trunk will update the JIRA. Thanks for pointing this out. Konst, Thats already been done in when tasks been assigned any queue. Thanks, Mayank
      Hide
      Mayank Bansal added a comment -

      Thanks Konst for your comments.

      Updated the patch with formatting issues.

      Thanks,
      Mayank

      Show
      Mayank Bansal added a comment - Thanks Konst for your comments. Updated the patch with formatting issues. Thanks, Mayank
      Hide
      Konstantin Shvachko added a comment -

      Patch looks good. Wanted to commit it, but mumak tests are failing.
      Run test-contrib target, you will see.

      Show
      Konstantin Shvachko added a comment - Patch looks good. Wanted to commit it, but mumak tests are failing. Run test-contrib target, you will see.
      Hide
      Konstantin Shvachko added a comment -

      Sorry my mistake. I had something else running on my box on port 50030, preventing SiulatorJobTracker to start for mumak. It's fine now. I'll commit the patch.
      Will remove unused variable counter in the new test.

      Show
      Konstantin Shvachko added a comment - Sorry my mistake. I had something else running on my box on port 50030, preventing SiulatorJobTracker to start for mumak. It's fine now. I'll commit the patch. Will remove unused variable counter in the new test.
      Hide
      Konstantin Shvachko added a comment -

      I just committed this. Thank you Mayank.

      Show
      Konstantin Shvachko added a comment - I just committed this. Thank you Mayank.
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-22-branch #108 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/108/)
      MAPREDUCE-4360. Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue. Contributed by Mayank Bansal. (Revision 1355514)

      Result = SUCCESS
      shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355514
      Files :

      • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
      • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/CapacityTaskScheduler.java
      • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueHierarchyBuilder.java
      • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueSchedulingContext.java
      • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred/TestContainerQueue.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #108 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/108/ ) MAPREDUCE-4360 . Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue. Contributed by Mayank Bansal. (Revision 1355514) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355514 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/CapacityTaskScheduler.java /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueHierarchyBuilder.java /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueSchedulingContext.java /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred/TestContainerQueue.java

        People

        • Assignee:
          Mayank Bansal
          Reporter:
          Mayank Bansal
        • Votes:
          0 Vote for this issue
          Watchers:
          6 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development