Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-893

Provide an ability to refresh queue configuration without restart.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Extended the framework's refresh-queue mechanism to support refresh of scheduler specific queue properties and implemented this refresh operation for some of the capacity scheduler properties. With this feature, one can refresh some of the capacity-scheduler's queue related properties - queue capacities, user-limits per queue, max map/reduce capacity and max-jobs per user to initialize while the system is running and without restarting JT. Even after this, some features like changing enable/disable priorities, adding/removing queues are not supported in capacity-scheduler.
      Show
      Extended the framework's refresh-queue mechanism to support refresh of scheduler specific queue properties and implemented this refresh operation for some of the capacity scheduler properties. With this feature, one can refresh some of the capacity-scheduler's queue related properties - queue capacities, user-limits per queue, max map/reduce capacity and max-jobs per user to initialize while the system is running and without restarting JT. Even after this, some features like changing enable/disable priorities, adding/removing queues are not supported in capacity-scheduler.

      Description

      While administering a cluster using multiple queues, administrators feel a need to refresh queue properties on the fly without needing to restart the JobTracker. This is partially supported for some properties such as queue ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is to extend the facility to refresh other queue properties as well, including scheduler properties.

      1. MAPREDUCE-893-7.patch
        159 kB
        rahul k singh
      2. MAPREDUCE-893-20090918.txt
        171 kB
        Vinod Kumar Vavilapalli
      3. MAPREDUCE-893-20090918.over.849.patch
        168 kB
        Vinod Kumar Vavilapalli
      4. MAPREDUCE-893-20090918.2.txt
        171 kB
        Vinod Kumar Vavilapalli
      5. MAPREDUCE-893-20090917.4.txt
        170 kB
        Vinod Kumar Vavilapalli
      6. MAPREDUCE-893-20090917.2.txt
        172 kB
        Vinod Kumar Vavilapalli
      7. MAPREDUCE-893-20090915.1.txt
        128 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hemanth Yamijala created issue -
          Vinod Kumar Vavilapalli made changes -
          Field Original Value New Value
          Assignee Vinod K V [ vinodkv ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by MAPREDUCE-860 [ MAPREDUCE-860 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by MAPREDUCE-860 [ MAPREDUCE-860 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by MAPREDUCE-861 [ MAPREDUCE-861 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090915.1.txt [ 12419615 ]
          Hemanth Yamijala made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090917.2.txt [ 12419863 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090917.4.txt [ 12419905 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090918.txt [ 12419968 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue incorporates MAPREDUCE-996 [ MAPREDUCE-996 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090918.2.txt [ 12420021 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-893-20090918.over.849.patch [ 12420024 ]
          rahul k singh made changes -
          Attachment MAPREDUCE-893-7.patch [ 12420052 ]
          rahul k singh made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note Extended the framework to refresh queue properties to support refresh of scheduler properties.
          Implemented the refresh operation for capacity scheduler properties like queue capacities.
          Resolution Fixed [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Release Note Extended the framework to refresh queue properties to support refresh of scheduler properties.
          Implemented the refresh operation for capacity scheduler properties like queue capacities.
          Extended the framework's refresh-queue mechanism to support refresh of scheduler specific queue properties and implemented this refresh operation for some of the capacity scheduler properties. With this feature, one can refresh some of the capacity-scheduler's queue related properties - queue capacities, user-limits per queue, max map/reduce capacity and max-jobs per user to initialize while the system is running and without restarting JT. Even after this, some features like changing enable/disable priorities, adding/removing queues are not supported in capacity-scheduler.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development