[HADOOP-16475] high number of calls to UserGroupInformation loginUserFromSubject creates huge number of ticket renewal thread which leads to Out of Memory

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.5
• Fix Version/s: 2.6.6, 2.8.3, 2.8.4
• Component/s: common

Environment

This issue has been observed all cloudera and hortonworks hadoop-common environments.

• Target Version/s: 2.8.5
• Flags: Important
• Assignee: Unassigned
• Reporter: Uday Kiran Reddy
• Created: 30/Jul/19 21:25
• Updated: 30/Jul/19 21:25

Description

We have an application which makes call to UserGroupInformation loginUserFromSubject multiple times depending on user requests. Each call is leading to create a new kerberos auto renewal thread (deamon) creation which holds ticket, configuration objects. This leads to constant memory growth of java process which eventually leads to process gets killed (out of memory). 

 

static void loginUserFromSubject(Subject subject)

{ .. loginUser.spawnAutoRenewalThreadForUserCreds(); --> this is created new auto renewal threads.   }

I think this is bug which needs to be fixed in hadoop common side. We should not created auto renewal thread for each request. Either give auto renewal process to the caller or make sure that there shouldn't be duplicate auto renewal for same user/subject. We are currently in a situation that we have huge number of threads but couldn't delete them or stop memory growth without restarting the services.