Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.1.1, 2.2.0
-
None
Description
STR:
- Set below config in spark-default.conf
spark.yarn.security.credentials.hbase.enabled true spark.hbase.connector.security.credentials.enabled false
- Set below config in hdfs-site.xml
'dfs.namenode.delegation.token.max-lifetime':'43200000' 'dfs.namenode.delegation.token.renew-interval':'28800000'
- Run HDFSWordcount streaming app in yarn-cluster mode for 25 hrs.
After 25 hours, noticing that HDFS Wordcount job is hitting HDFS_DELEGATION_TOKEN renewal issue.
17/06/28 10:49:47 WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired 17/06/28 10:49:47 WARN Client: Failed to cleanup staging dir hdfs://mycluster0/user/hrt_qa/.sparkStaging/application_1498539861056_0015 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554) at org.apache.hadoop.ipc.Client.call(Client.java:1498)