Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5220

Scheduling of OPPORTUNISTIC containers through YARN RM

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-beta1
    • resourcemanager
    • None
    • Hide
      This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
      This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without needing to enable distributed scheduling.
      Show
      This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion. This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without needing to enable distributed scheduling.

    Description

      In YARN-2882, we introduced the notion of OPPORTUNISTIC containers, along with the existing GUARANTEED containers of YARN.
      OPPORTUNISTIC containers are allowed to be queued at the NMs (YARN-2883), and are executed as long as there are available resources at the NM. Moreover, they are of lower priority than the GUARANTEED containers, that is, they can be preempted for a GUARANTEED container to start its execution.

      In YARN-2877, we introduced distributed scheduling in YARN, and enabled OPPORTUNISTIC containers to be scheduled exclusively by distributed schedulers.

      In this JIRA, we are proposing to extend the centralized YARN RM in order to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
      This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without the need to enable distributed scheduling.

      This JIRA is also related to YARN-1011 that introduces the over-commitment of resources, scheduling additional OPPORTUNISTIC containers to the NMs based on the currently used resources and not based only on the allocated resources.

      Attachments

        Issue Links

          Activity

            People

              kkaranasos Konstantinos Karanasos
              kkaranasos Konstantinos Karanasos
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: