[YARN-7765] [Atsv2] GSSException: No valid credentials provided - Failed to find any Kerberos tgt thrown by Timelinev2Client & HBaseClient in NM

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.0
• Fix Version/s: 3.1.0, 2.10.0, 3.0.1, 2.9.2
• Target Version/s: 3.1.0, 2.10.0, 3.0.1
• Assignee: Rohith Sharma K S
• Reporter: Sumana Sathish
• Created: 17/Jan/18 11:20
• Updated: 07/Nov/18 01:34
• Resolved: 28/Jan/18 22:19

Description

Secure cluster is deployed and all YARN services are started successfully. When application is submitted, app collectors which is started as aux-service throwing below exception. But this exception is NOT observed from RM TimelineCollector.

Cluster is deployed with Hadoop-3.0 and Hbase-1.2.6 secure cluster. All the YARN and HBase service are started and working perfectly fine. After 24 hours i.e when token lifetime is expired, HBaseClient in NM and HDFSClient in HMaster and HRegionServer started getting this error. After sometime, HBase daemons got shutdown. In NM, JVM didn't shutdown but none of the events got published.

2018-01-17 11:04:48,017 FATAL ipc.RpcClientImpl (RpcClientImpl.java:run(684)) - SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

cc :/ vrushalic varun_saxena