[YARN-9894] CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.2, 2.8.5, 3.2.1, 3.1.3
• Fix Version/s: 3.3.0, 3.2.2, 3.1.4, 2.10.1
• Component/s: capacity scheduler, test
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 11/Oct/19 16:07
• Updated: 18/Dec/19 21:49
• Resolved: 18/Dec/19 21:39

Description

I have developed a unit test based on the existing TestCapacitySchedulerPerf tests that will measure the performance of a configurable number of apps in a configurable number of queues. It will also test the performance of a cluster that has many queues but only a portion of them are active.

For example:

$ mvn test -Dtest=TestCapacitySchedulerPerf#testUserLimitThroughputWithManyQueues \
  -DRunCapacitySchedulerPerfTests=true
  -DNumberOfQueues=100 \
  -DNumberOfApplications=200 \
  -DPercentActiveQueues=100

• Parameters:
  • RunCapacitySchedulerPerfTests=true:
    Needed in order to trigger the test
  • NumberOfQueues
    Configurable number of queues
  • NumberOfApplications
    Total number of apps to run in the whole cluster, distributed evenly across all queues
  • PercentActiveQueues
    Percentage of the queues that contain active applications