Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6242 [Umbrella] Miscellaneous Scheduler Performance Improvements
  3. YARN-9276

Filter non-existent resource requests actively

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0
    • None
    • RM, scheduler
    • None

    Description

      In some scenarios, applications will request nonexistent resource name in this cluster, such as nonexistent hosts or racks.

      Obviously, we should filter or degrade these invaild resource requests actively.

      This is especially effective when HDFS and Yarn are deployed on different nodes, and the scheduling throughput of one of our clusters has improved by 5X.

      Attachments

        1. Cluster-Scheduler-Performance-5X-Promotion.png
          193 kB
          Zhaohui Xin
        2. YARN-9276.001.patch
          8 kB
          Zhaohui Xin

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7100 Provide options to skip adding resource request for data-local and rack-local respectively

          • Critical - Crashes, loss of data, severe memory leak.
          • Patch Available
          relates to

          Bug - A problem which impairs or prevents the functions of the product. YARN-8248 Job hangs when a job requests a resource that its queue does not have

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              uranus Zhaohui Xin
              uranus Zhaohui Xin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: