Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9894

CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      I have developed a unit test based on the existing TestCapacitySchedulerPerf tests that will measure the performance of a configurable number of apps in a configurable number of queues. It will also test the performance of a cluster that has many queues but only a portion of them are active.

      For example:
      $ mvn test -Dtest=TestCapacitySchedulerPerf#testUserLimitThroughputWithManyQueues \
        -DRunCapacitySchedulerPerfTests=true
        -DNumberOfQueues=100 \
        -DNumberOfApplications=200 \
        -DPercentActiveQueues=100
      
      • Parameters:
        • RunCapacitySchedulerPerfTests=true:
          Needed in order to trigger the test
        • NumberOfQueues
          Configurable number of queues
        • NumberOfApplications
          Total number of apps to run in the whole cluster, distributed evenly across all queues
        • PercentActiveQueues
          Percentage of the queues that contain active applications

      Attachments

        1. YARN-9894.001.patch
          12 kB
          epayne-old
        2. YARN-9894.002.patch
          13 kB
          Eric Payne

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            epayne Eric Payne
            epayne Eric Payne
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment