Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7149

Cross-queue preemption sometimes starves an underserved queue

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In branch 2 and trunk, I am consistently seeing some use cases where cross-queue preemption does not happen when it should. I do not see this in branch-2.8.

      Use Case:

        Size Minimum Container Size
      MyCluster 20 GB 0.5 GB
      Queue Name Capacity Absolute Capacity Minimum User Limit Percent (MULP) User Limit Factor (ULF)
      Q1 50% = 10 GB 100% = 20 GB 10% = 1 GB 2.0
      Q2 50% = 10 GB 100% = 20 GB 10% = 1 GB 2.0
      • User1 launches App1 in Q1 and consumes all resources (20 GB)
      • User2 launches App2 in Q2 and requests 10 GB
      • Note: containers are 0.5 GB.
      • Preemption monitor kills 2 containers (equals 1 GB) from App1 in Q1.
      • Capacity Scheduler assigns 2 containers (equals 1 GB) to App2 in Q2.
      • No more containers are ever preempted, even though Q2 is far underserved

        Attachments

        1. YARN-7149.002.patch
          7 kB
          Eric Payne
        2. YARN-7149.001.patch
          4 kB
          Eric Payne
        3. YARN-7149.demo.unit-test.patch
          3 kB
          Wangda Tan

          Activity

            People

            • Assignee:
              eepayne Eric Payne
              Reporter:
              eepayne Eric Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: