Major Description
In class AMDelegationTokenRenewer and ExecutorDelegationTokenUpdater, Spark will write and read the credentials.
But if we don't disable the fs.hdfs.impl.disable.cache, Spark will use cached FileSystem (which will use old token ) to upload or download file.
Then when the old token is expired, it can't gain the auth to get/put the hdfs.
(I only tested in a very short time with the configuration:
dfs.namenode.delegation.token.renew-interval=3min
dfs.namenode.delegation.token.max-lifetime=10min
I'm not sure whatever it matters.
)
