Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3409

Support Node Attribute functionality

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: api, client, RM
    • Labels:
      None
    • Release Note:
      With this feature task, Node Attributes is supported in YARN which will help user's to effectively use resources and assign to applications based on characteristics of each node's in the cluster.

      Description

      Specify only one label for each node (IAW, partition a cluster) is a way to determinate how resources of a special set of nodes could be shared by a group of entities (like teams, departments, etc.). Partitions of a cluster has following characteristics:

      • Cluster divided to several disjoint sub clusters.
      • ACL/priority can apply on partition (Only market team / marke team has priority to use the partition).
      • Percentage of capacities can apply on partition (Market team has 40% minimum capacity and Dev team has 60% of minimum capacity of the partition).

      Attributes are orthogonal to partition, they’re describing features of node’s hardware/software just for affinity. Some example of attributes:

      • glibc version
      • JDK version
      • Type of CPU (x86_64/i686)
      • Type of OS (windows, linux, etc.)

      With this, application can be able to ask for resource has (glibc.version >= 2.20 && JDK.version >= 8u20 && x86_64).

        Attachments

        1. 3409-apiChanges_v2.pdf (4).pdf
          71 kB
          Naganarasimha G R
        2. Constraint-Node-Labels-Requirements-Design-doc_v1.pdf
          153 kB
          Naganarasimha G R
        3. Node-Attributes-Requirements-Design-doc_v2.pdf
          143 kB
          Naganarasimha G R
        4. YARN-3409.WIP.001.patch
          91 kB
          Naganarasimha G R

          Issue Links

          1.
          CLI Proto Modifications to support Node Attributes Sub-task Resolved Naganarasimha G R
          2.
          Support CLI for Node Attributes Mapping Sub-task Resolved Naganarasimha G R
          3.
          Attribute Manager to store and provide node attributes in RM Sub-task Resolved Naganarasimha G R
          4.
          Support distributed node attributes Sub-task Resolved Bibin A Chundatt
          5.
          Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers Sub-task Resolved Weiwei Yang
          6.
          Update PB for prefix support of node attributes Sub-task Resolved Naganarasimha G R
          7.
          PB changes to carry node-attributes in NM heartbeat Sub-task Resolved Weiwei Yang
          8.
          Attach prefixes to different type of node attributes Sub-task Resolved LiangYe
          9.
          Validate Node Attributes from NM Sub-task Resolved Weiwei Yang
          10.
          Modify placement constraints to support node attributes Sub-task Resolved Sunil Govindan
          11.
          Node attributes documentation Sub-task Resolved Naganarasimha G R
          12.
          Node attributes reporting from NM to RM Sub-task Resolved Weiwei Yang
          13.
          Node Attribute store for storing and recovering attributes Sub-task Resolved Bibin A Chundatt
          14.
          Revisit NodeAttribute class structure Sub-task Resolved Naganarasimha G R
          15.
          NodeAttributeManager add/get API is not working properly Sub-task Resolved Weiwei Yang
          16.
          Refactor FSNodeLabelStore code for Node Attributes store support Sub-task Resolved Bibin A Chundatt
          17.
          Node attribute prefix definition and validation Sub-task Resolved Weiwei Yang
          18.
          Expose Node Attributes info via RM nodes REST API Sub-task Resolved Weiwei Yang
          19.
          Support configuration based Node Attribute provider Sub-task Resolved Weiwei Yang
          20.
          Support API interface to query cluster attributes and attribute to nodes Sub-task Resolved Bibin A Chundatt
          21.
          Add CLI interface to query node attributes Sub-task Resolved Bibin A Chundatt
          22.
          CLI Integration with NodeAttributesManagerImpl Sub-task Resolved Naganarasimha G R
          23.
          Add API to fetch node to attribute mapping Sub-task Resolved Bibin A Chundatt
          24.
          Fix TestRMWebServicesNodes test failure Sub-task Resolved Bibin A Chundatt
          25.
          Modify distributedshell to support Node Attributes Sub-task Resolved Naganarasimha G R
          26.
          Node attribute manager logs are flooding RM logs Sub-task Resolved Weiwei Yang
          27.
          Allow dot in attribute values Sub-task Resolved Bibin A Chundatt
          28.
          Merge related work for YARN-3409 Sub-task Resolved Unassigned
          29.
          Relax NE node-attribute check when attribute doesn't exist on a node Sub-task Resolved Sunil Govindan
          30.
          Fix jenkins issues for Node Attributes branch Sub-task Resolved Sunil Govindan
          31.
          Clear node attribute path after each test run Sub-task Resolved Bibin A Chundatt

            Activity

              People

              • Assignee:
                Naganarasimha Naganarasimha G R
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                2 Vote for this issue
                Watchers:
                46 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: