[HADOOP-11151] Automatically refresh auth token and retry on auth failure

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.0
• Component/s: security
• Assignee: Arun Suresh
• Reporter: zhubin
• Created: 29/Sep/14 08:18
• Updated: 01/Dec/14 03:11
• Resolved: 03/Oct/14 03:03

Description

Enable CFS and KMS service in the cluster, initially it worked to put/copy file into encryption zone. But after a while (might be one day), it fails to put/copy file into the encryption zone with the error
java.util.concurrent.ExecutionException: java.io.IOException: HTTP status [403], message [Forbidden]

The kms.log shows below
AbstractDelegationTokenSecretManager - Updating the current master key for generating delegation tokens
2014-09-29 13:18:46,599 WARN AuthenticationFilter - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2014-09-29 13:18:46,599 WARN AuthenticationFilter - Authentication exception: Anonymous requests are disallowed
org.apache.hadoop.security.authentication.client.AuthenticationException: Anonymous requests are disallowed
at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:184)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:331)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
at org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: HADOOP-11113 Namenode not able to reconnect to KMS after KMS restart