[OOZIE-2413] Kerberos credentials can expire if the KDC is slow to respond

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: 4.3.0
• Component/s: security
• Assignee: Robert Kanter
• Reporter: Robert Kanter
• Created: 04/Dec/15 01:51
• Updated: 02/Dec/16 21:04
• Resolved: 07/Dec/15 18:52

Description

We've seen some very rare cases where Oozie gets a Kerberos error when trying to get delegation tokens via the Credentials mechanism (e.g. getting HS2 delegation tokens).

We finally narrowed it down to slow KDC responses, so Oozie's Kerberos credentials have expired when it tries to get the delegation token. The reason we don't see this with Hadoop clients (DFSClient for HDFS, JobClient for MR, etc) is because they call UserGroupInformation#checkTGTAndReloginFromKeytab() before trying to connect.

We should do a similar fix by calling UserGroupInformation#checkTGTAndReloginFromKeytab() before using a Credentials implementation.