[YARN-4336] YARN NodeManager - Container Initialization - Excessive load on NSS/LDAP

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.6.0, 2.7.0, 2.6.1, 2.7.1

Environment

NSS w/ SSSD or Dell/Quest - VASD

• Target Version/s: 2.8.0
• Release Note: Tactical workaround
• Assignee: Greg Senia
• Reporter: Greg Senia
• Created: 06/Nov/15 15:28
• Updated: 19/Mar/16 07:10
• Resolved: 06/Nov/15 16:32

Description

Hi folks after performing some debug for our Unix Engineering and Active Directory teams it was discovered that on YARN Container Initialization a call via Hadoop Common AccessControlList.java:

for(String group: ugi.getGroupNames()) {
if (groups.contains(group))

{ return true; }
}

Unfortunately with the security call to check access on "appattempt_XXXXXXXXXXXXX_XXXXX_XXXXX" will always return false but will make unnecessary calls to NameSwitch service on linux which will call things like SSSD/Quest VASD which will then initiate LDAP calls looking for non existent userid's causing excessive load on LDAP.

For now our tactical work around is as follows:



Example of VASD Debug log showing the lookups for one task attempt 32 of them:
/**
* Checks if a user represented by the provided {@link UserGroupInformation}
* is a member of the Access Control List
* @param ugi UserGroupInformation to check if contained in the ACL
* @return true if ugi is member of the list
*/
public final boolean isUserInList(UserGroupInformation ugi) {
if (allAllowed || users.contains(ugi.getShortUserName())) { return true; } else {
String patternString = "^appattempt_\\d+\\d+
d+$";

Pattern pattern = Pattern.compile(patternString);

Matcher matcher = pattern.matcher(ugi.getShortUserName());
boolean matches = matcher.matches();
if (matches) { LOG.debug("Bailing !! AppAttempt Matches DONOT call UGI FOR GROUPS!!");; return false; }


for(String group: ugi.getGroupNames()) {
if (groups.contains(group)) { return true; }

}
}
return false;
}

public boolean isUserAllowed(UserGroupInformation ugi)

{ return isUserInList(ugi); }

One task:
Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>
Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM> with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, base=<>, scope=<sub>

Issue Links

• duplicates: YARN-3452 Bogus token usernames cause many invalid group lookups