[TWILL-106] HDFS delegation token is not being refreshed properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.4.0-incubating
• Component/s: core
• Assignee: Unassigned
• Reporter: Poorna Chandra
• Created: 28/Oct/14 01:48
• Updated: 09/Nov/14 09:40
• Resolved: 09/Nov/14 09:40

Description

We have a Twill app that runs in a secure Hadoop cluster. The app starts up fine, and runs for a day. I can see in logs that say secure store was updated regularly. However, after a day I see exceptions that say "token (HDFS_DELEGATION_TOKEN token 4287 for yarn) can't be found in cache".

Exception:
-------------
2014-10-23T04:12:42,101Z ERROR c.c.t.TransactionManager [cdap-secure120-1000.dev.continuuity.net] [tx-snapshot] TransactionManager:abortService(TransactionManager.java:594) - Aborting transaction manager due to: Snapshot (timestamp 1414037562088) failed due to: token (HDFS_DELEGATION_TOKEN token 4287 for yarn) can't be found in cache
org.apache.hadoop.ipc.RemoteException: token (HDFS_DELEGATION_TOKEN token 4287 for yarn) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
...