Hadoop YARN
  1. Hadoop YARN
  2. YARN-18

Configurable Hierarchical Topology for YARN

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.

      1. YARN-18-v8.0.patch
        51 kB
        Wenwu Peng
      2. YARN-18-v7.patch
        144 kB
        Junping Du
      3. YARN-18-v7.3.patch
        143 kB
        Junping Du
      4. YARN-18-v7.2.patch
        144 kB
        Junping Du
      5. YARN-18-v7.1.patch
        144 kB
        Junping Du
      6. YARN-18-v6.patch
        99 kB
        Junping Du
      7. YARN-18-v6.4.patch
        99 kB
        Junping Du
      8. YARN-18-v6.3.patch
        99 kB
        Junping Du
      9. YARN-18-v6.2.patch
        99 kB
        Junping Du
      10. YARN-18-v6.1.patch
        99 kB
        Junping Du
      11. YARN-18-v5.patch
        95 kB
        Junping Du
      12. YARN-18-v5.1.patch
        95 kB
        Junping Du
      13. YARN-18-v4.patch
        80 kB
        Junping Du
      14. YARN-18-v4.3.patch
        88 kB
        Junping Du
      15. YARN-18-v4.2.patch
        89 kB
        Junping Du
      16. YARN-18-v4.1.patch
        89 kB
        Junping Du
      17. YARN-18-v3.patch
        51 kB
        Junping Du
      18. YARN-18-v3.2.patch
        47 kB
        Junping Du
      19. YARN-18-v3.1.patch
        51 kB
        Junping Du
      20. YARN-18-v2.patch
        52 kB
        Junping Du
      21. YARN-18.v8.1.patch
        50 kB
        Wenwu Peng
      22. YARN-18.patch
        52 kB
        Junping Du
      23. Pluggable topologies with NodeGroup for YARN.pdf
        186 kB
        Junping Du
      24. MAPREDUCE-4309-v7.patch
        53 kB
        Junping Du
      25. MAPREDUCE-4309-v6.patch
        53 kB
        Junping Du
      26. MAPREDUCE-4309-v5.patch
        53 kB
        Junping Du
      27. MAPREDUCE-4309-v4.patch
        39 kB
        Junping Du
      28. MAPREDUCE-4309-v3.patch
        39 kB
        Junping Du
      29. MAPREDUCE-4309-v2.patch
        29 kB
        Junping Du
      30. MAPREDUCE-4309.patch
        61 kB
        Junping Du
      31. HierachicalTopologyForYARNr1.pdf
        276 kB
        Junping Du
      32. HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
        61 kB
        Junping Du

        Issue Links

          Activity

          Junping Du created issue -
          Junping Du made changes -
          Field Original Value New Value
          Assignee Junping Du [ djp ]
          Junping Du made changes -
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Parent HADOOP-8468 [ 12559114 ]
          Issue Type Sub-task [ 7 ] Bug [ 1 ]
          Junping Du made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-8474 MAPREDUCE-4309
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 1.0.0 [ 12318244 ]
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Junping Du made changes -
          Link This issue is part of HADOOP-8468 [ HADOOP-8468 ]
          Junping Du made changes -
          Link This issue blocks MAPREDUCE-4310 [ MAPREDUCE-4310 ]
          Junping Du made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309.patch [ 12535320 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v2.patch [ 12535376 ]
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v3.patch [ 12535504 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v4.patch [ 12535513 ]
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v5.patch [ 12536840 ]
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v6.patch [ 12537322 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v7.patch [ 12537557 ]
          Arun C Murthy made changes -
          Issue Type Bug [ 1 ] New Feature [ 2 ]
          Arun C Murthy made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Key MAPREDUCE-4309 YARN-18
          Workflow no-reopen-closed, patch-avail [ 12671853 ] jira [ 12720946 ]
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Gavin made changes -
          Workflow jira [ 12720946 ] no-reopen-closed, patch-avail [ 12722153 ]
          Junping Du made changes -
          Attachment YARN-18.patch [ 12554113 ]
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.0.3-alpha [ 12323272 ]
          Labels features
          Junping Du made changes -
          Attachment YARN-18-v2.patch [ 12554120 ]
          Junping Du made changes -
          Attachment YARN-18-v3.patch [ 12571919 ]
          Junping Du made changes -
          Attachment YARN-18-v3.1.patch [ 12572014 ]
          Junping Du made changes -
          Attachment YARN-18-v3.2.patch [ 12572023 ]
          Junping Du made changes -
          Attachment YARN-18-v4.patch [ 12573355 ]
          Junping Du made changes -
          Attachment YARN-18-v4.1.patch [ 12573471 ]
          Junping Du made changes -
          Attachment YARN-18-v4.2.patch [ 12573476 ]
          Junping Du made changes -
          Attachment YARN-18-v4.3.patch [ 12574124 ]
          Junping Du made changes -
          Attachment YARN-18-v5.patch [ 12575479 ]
          Junping Du made changes -
          Attachment YARN-18-v5.1.patch [ 12575487 ]
          Junping Du made changes -
          Junping Du made changes -
          Attachment YARN-18-v6.patch [ 12581362 ]
          Gavin made changes -
          Link This issue blocks YARN-19 [ YARN-19 ]
          Gavin made changes -
          Link This issue is depended upon by YARN-19 [ YARN-19 ]
          Gavin made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-8469 [ HADOOP-8469 ]
          Junping Du made changes -
          Attachment YARN-18-v6.1.patch [ 12581468 ]
          Junping Du made changes -
          Attachment YARN-18-v6.2.patch [ 12581846 ]
          Junping Du made changes -
          Attachment YARN-18-v6.3.patch [ 12582025 ]
          Junping Du made changes -
          Attachment YARN-18-v6.4.patch [ 12582077 ]
          Junping Du made changes -
          Attachment YARN-18-v7.patch [ 12586885 ]
          Junping Du made changes -
          Attachment YARN-18-v7.1.patch [ 12586893 ]
          Junping Du made changes -
          Attachment YARN-18-v7.2.patch [ 12586900 ]
          Junping Du made changes -
          Attachment YARN-18-v7.3.patch [ 12587041 ]
          Junping Du made changes -
          Attachment HierachicalTopologyForYARNr1.pdf [ 12594353 ]
          Junping Du made changes -
          Summary Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology Configurable Hierarchical Topology for YARN
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Description There are several classes in YARN’s container assignment and task scheduling algorithms that relate to data locality which were updated to give preference to running a container on other locality besides node-local and rack-local (like nodegroup-local). This propose to make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class ScheduledRequests was made a package level class to it would be easier to create a subclass, ScheduledRequestsWithNodeGroup. Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.
          Junping Du made changes -
          Link This issue blocks HADOOP-10512 [ HADOOP-10512 ]
          Wenwu Peng made changes -
          Attachment YARN-18-v8-1.patch [ 12658610 ]
          Wenwu Peng made changes -
          Attachment YARN-18-v8-1.patch [ 12658610 ]
          Wenwu Peng made changes -
          Attachment YARN-18-v8.0.patch [ 12658611 ]
          Wenwu Peng made changes -
          Attachment YARN-18.v8.1.patch [ 12658617 ]

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated:

                Development