Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7812 Improvements to Rich Placement Constraints in YARN
  3. YARN-8013

Support application tags when defining application namespaces for placement constraints

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.1
    • Component/s: None
    • Labels:
      None

      Description

      YARN-1461 adds the concept of Application Tags to Yarn applications. In particular, a user is able to annotate application with multiple tags to classify apps. We can leverage this to define application namespaces based on application tags for placement constraints.

      Below is a typical use case.

      There are a lot of TF jobs running on Yarn, and some of them are consuming resources heavily. So we want to limit number of PS on each node for such BIG players but ignore those SMALL ones. To achieve this, we can do following steps:

      1. Add application tag "big-tf" to these big TF jobs
      2. For each PS request, we add "ps" source tag and map it to constraint "notin, node, tensorflow/ps" or "cardinality, node, tensorflow/ps, 0, 2" for finer grained controls.

        Attachments

        1. YARN-8013.001.patch
          28 kB
          Weiwei Yang
        2. YARN-8013.002.patch
          27 kB
          Weiwei Yang
        3. YARN-8013.003.patch
          28 kB
          Weiwei Yang
        4. YARN-8013.004.patch
          28 kB
          Weiwei Yang
        5. YARN-8013.005.patch
          37 kB
          Weiwei Yang
        6. YARN-8013.006.patch
          50 kB
          Weiwei Yang
        7. YARN-8013.007.patch
          50 kB
          Weiwei Yang

          Activity

            People

            • Assignee:
              cheersyang Weiwei Yang
              Reporter:
              cheersyang Weiwei Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: