[HADOOP-17079] Optimize UGI#getGroups by adding UGI#getGroupsSet

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: build
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed

Release Note

Added a UserGroupMapping#getGroupsSet() API and deprecate UserGroupMapping#getGroups.

The UserGroupMapping#getGroups() can be expensive as it involves Set->List conversion. For user with large group membership (i.e., > 1000 groups), we recommend using getGroupSet to avoid the conversion and fast membership look up.

• Assignee: Xiaoyu Yao
• Reporter: Xiaoyu Yao
• Created: 21/Jun/20 02:28
• Updated: 11/Feb/24 06:32
• Resolved: 09/Jul/20 18:35

Description

UGI#getGroups has been optimized with HADOOP-13442 by avoiding the List->Set->List conversion. However the returned list is not optimized to contains lookup, especially the user's group membership list is huge (thousands+) . This ticket is opened to add a UGI#getGroupsSet and use Set#contains() instead of List#contains() to speed up large group look up while minimize List->Set conversions in Groups#getGroups() call.

Issue Links

• breaks: HADOOP-17505 public interface GroupMappingServiceProvider needs default impl for getGroupsSet()
• causes: HADOOP-17467 netgroup-user is not added to Groups.cache
• causes: YARN-10350 TestUserGroupMappingPlacementRule fails
• incorporates: HADOOP-17487 Replace deprecated getGroups with getGroupsSet
• is related to: HADOOP-13442 Optimize UGI group lookups
• relates to: HADOOP-17485 port UGI#getGroupsSet optimizations into 2.10
• links to: GitHub Pull Request #2631

Sub-Tasks

• Port HADOOP-17079, HADOOP-17505 to branch-3.3: Reopened