[SPARK-2541] Standalone mode can't access secure HDFS anymore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.0.0, 1.0.1
• Component/s: Deploy
• Labels: bulk-closed
• Target Version/s: 1.0.3, 1.1.0
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 17/Jul/14 01:08
• Updated: 06/Jun/19 14:01
• Resolved: 06/Jun/19 14:01

Description

In spark 0.9.x you could access secure HDFS from Standalone deploy, that doesn't work in 1.X anymore.

It looks like the issues is in SparkHadoopUtil.runAsSparkUser. Previously it wouldn't do the doAs if the currentUser == user. Not sure how it affects when the daemons run as a super user but SPARK_USER is set to someone else.

Issue Links

• duplicates: SPARK-3438 Support for accessing secured HDFS in Standalone Mode
• relates to: SPARK-3438 Support for accessing secured HDFS in Standalone Mode