[YARN-10889] [Umbrella] Queue Creation in Capacity Scheduler - Tech debts

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0, 3.5.0
• Component/s: capacity scheduler
• Labels: scheduler
• Target Version/s: 3.4.0
• Assignee: Benjamin Teke
• Reporter: Szilard Nemeth
• Created: 18/Aug/21 09:18
• Updated: 27/Jan/24 00:25
• Resolved: 26/Jan/24 13:45

Description

Follow-up of YARN-10496

Issue Links

• is related to: YARN-11652 [Umbrella] Follow-up after YARN-10888/YARN-10889

Sub-Tasks

• Follow-up to YARN-10504: Resolved
• Consider legacy auto queue creation absolute resource template to avoid rounding errors: Resolved
• Make auto queue creation maximum allowed depth configurable: Resolved
• Encapsulate queue config modes: Resolved
• Introduce QueuePath class: Resolved
• TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario: Resolved
• Remove unnecessary fields from AbstractCSQueue or group fields by feature if possible: Resolved
• Investigate if AbstractCSQueue#configuredNodeLabels vs. QueueCapacities#getExistingNodeLabels holds the same data: Resolved
• AbstractCSQueue: Create QueueConfig object for generic queue-specific fields: Resolved
• Minimize usages of AbstractCSQueue#csContext: Resolved
• Investigate: Why AbstractCSQueue#authorizer is constructed for each queue: Resolved
• AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code: Resolved
• AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic: Resolved
• AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map: Resolved
• AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic: Resolved
• AbstractCSQueue: Group preemption methods and fields into a separate class: Resolved
• Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue: Resolved
• AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources: Resolved
• Simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges: Resolved
• Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy: Resolved
• Simplify method: CapacitySchedulerQueueManager#parseQueue: Resolved
• Remove LeafQueue#scheduler field: Resolved
• Create a dedicated class for Node Labels: Resolved
• Investigation: Verify if legacy AQC works as documented: Resolved
• Investigate if creating separate classes for Dynamic Leaf / Dynamic Parent queues makes sense: Resolved
• Clean up CapacityScheduler#initScheduler: Resolved
• Simplify AbstractCSQueue#setupQueueConfigs: Resolved
• Test validation after YARN-10504 and YARN-10506: Check if modified test expectations are correct or not: Resolved
• Explain assertion literals in testcases of CapacityScheduler and related test classes: Resolved
• Do not use a separate config in legacy CS AQC: Resolved
• Move AbstractCSQueue fields to separate objects that are tracking usage: Resolved
• AbstractCSQueue: Create separate class for encapsulating Min / Max Resource: Resolved
• AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity: Resolved
• Add javadoc to all methods of AbstractCSQueue: Resolved
• AbstractCSQueue: Create separate class for constructing Queue API objects: Resolved
• Simplify AbstractCSQueue#initializeQueueState: Resolved
• Rename SchedulerQueue#activeQueue to activateQueue: Resolved
• Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method: Resolved
• Code cleanup in QueueCapacities: Resolved
• CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class: Resolved
• Make CapacityScheduler#getOrCreateQueueFromPlacementContext easier to comprehend: Resolved
• Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf: Resolved
• Extract common method of two that check if preemption disabled in CSQueuePreemption: Resolved
• Extract test queues and related methods from TestCapacityScheduler: Resolved
• TestCapacityScheduler: reuse appHelper where feasible: Resolved
• Do not extend from CapacitySchedulerTestBase when not needed: Resolved
• Split TestCapacityScheduler by test categories: Resolved
• Use correct configuration for Group service init in CSMappingPlacementRule: Resolved
• Replace all occurences of queuePath with the new QueuePath class: Resolved
• Follow-up changes for YARN-10904: Resolved
• Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration: Resolved
• Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy: Resolved
• Unify node label access in queues: Resolved
• Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality: Resolved
• Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched: Resolved
• Do not inherit from TestRMWebServicesCapacitySched: Resolved
• Fix testQueueSubmitWithACL* tests in TestAppManager: Resolved
• Replace all occurences of queuePath with the new QueuePath class - followup: Resolved
• Clean up checkstyle warnings from YARN-11024/10907/10929: Resolved
• Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality: Resolved