Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5764

NUMA awareness support for launching containers

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0, 3.2.0
    • Component/s: nodemanager, yarn
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The purpose of this feature is to improve Hadoop performance by minimizing costly remote memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific NUMA node and all subsequent memory allocations will be served by the same node, reducing remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.

        Attachments

        1. NUMA Awareness for YARN Containers.pdf
          288 kB
          Devaraj Kavali
        2. NUMA Performance Results.pdf
          594 kB
          Devaraj Kavali
        3. YARN-5764-v0.patch
          45 kB
          Devaraj Kavali
        4. YARN-5764-v1.patch
          45 kB
          Devaraj Kavali
        5. YARN-5764-v10.patch
          65 kB
          Devaraj Kavali
        6. YARN-5764-v11.patch
          64 kB
          Devaraj Kavali
        7. YARN-5764-v2.patch
          45 kB
          Devaraj Kavali
        8. YARN-5764-v3.patch
          47 kB
          Devaraj Kavali
        9. YARN-5764-v4.patch
          63 kB
          Devaraj Kavali
        10. YARN-5764-v5.patch
          63 kB
          Devaraj Kavali
        11. YARN-5764-v6.patch
          62 kB
          Devaraj Kavali
        12. YARN-5764-v7.patch
          62 kB
          Devaraj Kavali
        13. YARN-5764-v8.patch
          62 kB
          Devaraj Kavali
        14. YARN-5764-v9.patch
          62 kB
          Devaraj Kavali

          Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-7033 Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. YARN-8168 Add support in Winutils for reporting CPU cores in all CPU groups, and aggregate kernel time, idle time and user time for all CPU groups

          • Major - Major loss of function.
          • Patch Available

            Activity

              People

              • Assignee:
                devaraj Devaraj Kavali
                Reporter:
                olasoji Olasoji
              • Votes:
                0 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: