[SPARK-14602] [YARN+Windows] Setting SPARK_YARN_CACHE_FILES exceeds command line length limit on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Deploy, Spark Core, Spark Submit, Windows, YARN

Environment

YARN on Windows

• Target Version/s: 2.0.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Sebastian Kochman
• Created: 13/Apr/16 18:00
• Updated: 17/May/20 18:17
• Resolved: 20/Apr/16 23:57

Description

After change https://issues.apache.org/jira/browse/SPARK-11157, which removed a single large Spark assembly in favor of multiple small jars, when you try to submit a Spark app to YARN on Windows (using spark-submit.cmd), the app fails with the following error:

Diagnostics: The command line has a length of 12046 exceeds maximum allowed length of 8191. Command starts with: @set SPARK_YARN_CACHE_FILES=[...]/.sparkStaging/application_[...] Failing this attempt. Failing the application.

So basically a large number of jars needed for staging in YARN causes exceeding Windows command line length limit.

Please see more details in the discussion here:
https://issues.apache.org/jira/browse/SPARK-11157?focusedCommentId=15238151&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15238151

Issue Links

• links to: [Github] Pull Request #12487 (vanzin)