[SPARK-26895] When running spark 2.3 as a proxy user (--proxy-user), SparkSubmit fails to resolve globs owned by target user

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.2, 2.4.0
• Fix Version/s: 2.3.4, 2.4.4, 3.0.0
• Component/s: Spark Core
• Assignee: Alessandro Bellina
• Reporter: Alessandro Bellina
• Created: 15/Feb/19 19:43
• Updated: 22/Aug/19 07:40
• Resolved: 22/Feb/19 19:15

Description

We are resolving globs in SparkSubmit here (by way of prepareSubmitEnvironment) without first going into a doAs:

https://github.com/apache/spark/blob/6c18d8d8079ac4d2d6dc7539601ab83fc5b51760/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L143

Without first entering a doAs, as done here:

https://github.com/apache/spark/blob/6c18d8d8079ac4d2d6dc7539601ab83fc5b51760/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L151

So when running spark-submit with --proxy-user, and for example --archives, it will fail to launch unless the location of the archive is open to the user that executed spark-submit.

Issue Links

• is caused by: SPARK-21012 Support glob path for resources adding to Spark
• is duplicated by: SPARK-28417 Spark Submit does not use Proxy User Credentials to Resolve Path for Resources
• links to: GitHub Pull Request #23806
• links to: GitHub Pull Request #23911
• links to: GitHub Pull Request #25516
• links to: GitHub Pull Request #25517