[SPARK-3778] newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.3.0
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 03/Oct/14 12:57
• Updated: 26/May/15 06:54
• Resolved: 03/Feb/15 06:46

Description

The newAPIHadoopRDD routine doesn't properly add the credentials to the conf to be able to access secure hdfs.

Note that newAPIHadoopFile does handle these because the org.apache.hadoop.mapreduce.Job automatically adds it for you.

Issue Links

• is duplicated by: SPARK-7110 when use saveAsNewAPIHadoopFile, sometimes it throws "Delegation Token can be issued only with kerberos or web authentication"
• links to: [Github] Pull Request #2676 (tgravescs)
• links to: [Github] Pull Request #4292 (tgravescs)