Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10889

[Umbrella] Queue Creation in Capacity Scheduler - Tech debts

    XMLWordPrintableJSON

Details

    Description

      Follow-up of YARN-10496

      Attachments

        Issue Links

          1.
          Follow-up to YARN-10504 Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          2.
          Consider legacy auto queue creation absolute resource template to avoid rounding errors Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          3.
          Make auto queue creation maximum allowed depth configurable Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          4.
          Encapsulate queue config modes Sub-task Resolved Andras Gyori  
          5.
          Introduce QueuePath class Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h
          6.
          TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          7.
          Remove unnecessary fields from AbstractCSQueue or group fields by feature if possible Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10h 50m
          8.
          Investigate if AbstractCSQueue#configuredNodeLabels vs. QueueCapacities#getExistingNodeLabels holds the same data Sub-task Resolved Peter Szucs  
          9.
          AbstractCSQueue: Create QueueConfig object for generic queue-specific fields Sub-task Resolved Szilard Nemeth  
          10.
          Minimize usages of AbstractCSQueue#csContext Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 40m
          11.
          Investigate: Why AbstractCSQueue#authorizer is constructed for each queue Sub-task Resolved Szilard Nemeth  
          12.
          AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          13.
          AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          14.
          AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          15.
          AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          16.
          AbstractCSQueue: Group preemption methods and fields into a separate class Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 50m
          17.
          Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          18.
          AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          19.
          Simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          20.
          Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3.5h
          21.
          Simplify method: CapacitySchedulerQueueManager#parseQueue Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h
          22.
          Remove LeafQueue#scheduler field Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          23.
          Create a dedicated class for Node Labels Sub-task Resolved Unassigned  
          24.
          Investigation: Verify if legacy AQC works as documented Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          25.
          Investigate if creating separate classes for Dynamic Leaf / Dynamic Parent queues makes sense Sub-task Resolved András Győri  
          26.
          Clean up CapacityScheduler#initScheduler Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          27.
          Simplify AbstractCSQueue#setupQueueConfigs Sub-task Resolved Benjamin Teke  
          28.
          Test validation after YARN-10504 and YARN-10506: Check if modified test expectations are correct or not Sub-task Resolved Peter Szucs  
          29.
          Explain assertion literals in testcases of CapacityScheduler and related test classes Sub-task Resolved Tamas Domok  
          30.
          Do not use a separate config in legacy CS AQC Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 10m
          31.
          Move AbstractCSQueue fields to separate objects that are tracking usage Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h
          32.
          AbstractCSQueue: Create separate class for encapsulating Min / Max Resource Sub-task Resolved Szilard Nemeth  
          33.
          AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          34.
          Add javadoc to all methods of AbstractCSQueue Sub-task Resolved András Győri

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          35.
          AbstractCSQueue: Create separate class for constructing Queue API objects Sub-task Resolved Peter Szucs  
          36.
          Simplify AbstractCSQueue#initializeQueueState Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          37.
          Rename SchedulerQueue#activeQueue to activateQueue Sub-task Resolved Adam Antal

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          38.
          Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          39.
          Code cleanup in QueueCapacities Sub-task Resolved Adam Antal

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          40.
          CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          41.
          Make CapacityScheduler#getOrCreateQueueFromPlacementContext easier to comprehend Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          42.
          Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          43.
          Extract common method of two that check if preemption disabled in CSQueuePreemption Sub-task Resolved Peter Szucs  
          44.
          Extract test queues and related methods from TestCapacityScheduler Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          45.
          TestCapacityScheduler: reuse appHelper where feasible Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          46.
          Do not extend from CapacitySchedulerTestBase when not needed Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 10m
          47.
          Split TestCapacityScheduler by test categories Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          48.
          Use correct configuration for Group service init in CSMappingPlacementRule Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h
          49.
          Replace all occurences of queuePath with the new QueuePath class Sub-task Resolved Tibor Kovács

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 40m
          50.
          Follow-up changes for YARN-10904 Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          51.
          Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration Sub-task Resolved Szilard Nemeth

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          52.
          Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          53.
          Unify node label access in queues Sub-task Resolved Andras Gyori

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          54.
          Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          55.
          Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          56.
          Do not inherit from TestRMWebServicesCapacitySched Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          57.
          Fix testQueueSubmitWithACL* tests in TestAppManager Sub-task Resolved Tamas Domok

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          58.
          Replace all occurences of queuePath with the new QueuePath class - followup Sub-task Resolved Peter Szucs  
          59.
          Clean up checkstyle warnings from YARN-11024/10907/10929 Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          60.
          Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality Sub-task Resolved Susheel Gupta  

          Activity

            People

              bteke Benjamin Teke
              snemeth Szilard Nemeth
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 102h 50m
                  102h 50m