[MAPREDUCE-6876] FileInputFormat.listStatus should not fetch delegation tokens

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Michael Gummelt
• Created: 13/Apr/17 23:38
• Updated: 14/Apr/17 21:01

Description

FileInputFormat.listStatus fetches delegation tokens: https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213

AFAICT, this is unnecessary. listStatus doesn't delegate those tokens to another process. This is causing issues described in the attached Spark Kerberos ticket, because TokenCache.obtainTokensForNameNodes, which is used to fetch the delegation tokens, assumes that certain MapReduce configuration variables are set, which isn't true in the Spark calling code. This is a separate problem, but nonetheless it wouldn't have arisen if listStatus weren't fetching delegation tokens.

Issue Links

• relates to: SPARK-20328 HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs