[YARN-10958] Use correct configuration for Group service init in CSMappingPlacementRule

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Szilard Nemeth
• Reporter: Peter Bacsko
• Created: 16/Sep/21 12:29
• Updated: 12/Feb/24 06:27
• Resolved: 20/Oct/21 12:24

Description

There is a potential problem in CSMappingPlacementRule.java:

    if (groups == null) {
      groups = Groups.getUserToGroupsMappingService(conf);
    }

The problem is, we're supposed to pass scheduler.getConf(). The "conf" object is the config for capacity scheduler, which does not include the property which selects the group service provider. Therefore, the current code just works by chance, because Group mapping service is already initialized at this point. See the original fix in YARN-10053.

Also, need a unit test to verify it.

Idea:

1. Create a Configuration object in which the property "hadoop.security.group.mapping" refers to an existing a test implementation.
2. Add a new method to Groups which nulls out the singleton instance, eg. Groups.reset().
3. Create a mock CapacityScheduler where getConf() and getConfiguration() contain different settings for "hadoop.security.group.mapping". Since getConf() is the service config, this should return the config object created in step #1.
4. Create an instance of CSMappingPlacementRule with a single primary group rule.
5. Run the placement evaluation.
6. Expected: returned queue matches what is supposed to be coming from the test group mapping service ("testuser" --> "testqueue").
7. Modify "hadoop.security.group.mapping" in the config object created in step #1.
8. Call Groups.refresh() which changes the group mapping ("testuser" --> "testqueue2"). This requires that the test group mapping service implement GroupMappingServiceProvider.cacheGroupsRefresh().
9. Create a new instance of CSMappingPlacementRule.
10. Run the placement evaluation again
11. Expected: with the same user, the target queue has changed.

This looks convoluted, but these steps make sure that:

1. CSMappingPlacementRule will force the initialization of groups.
2. We select the correct configuration for group service init.
3. We don't create a new Groups instance if the singleton is initialized, so we cover the original problem described in YARN-10597.

Issue Links

• links to: GitHub Pull Request #3560