Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14602

[YARN+Windows] Setting SPARK_YARN_CACHE_FILES exceeds command line length limit on Windows

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Deploy, Spark Submit, Windows, YARN
    • Labels:
      None
    • Environment:

      YARN on Windows

    • Target Version/s:

      Description

      After change https://issues.apache.org/jira/browse/SPARK-11157, which removed a single large Spark assembly in favor of multiple small jars, when you try to submit a Spark app to YARN on Windows (using spark-submit.cmd), the app fails with the following error:

      Diagnostics: The command line has a length of 12046 exceeds maximum allowed length of 8191. Command starts with: @set SPARK_YARN_CACHE_FILES=[...]/.sparkStaging/application_[...] Failing this attempt. Failing the application.

      So basically a large number of jars needed for staging in YARN causes exceeding Windows command line length limit.

      Please see more details in the discussion here:
      https://issues.apache.org/jira/browse/SPARK-11157?focusedCommentId=15238151&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15238151

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              sebastko Sebastian Kochman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: