Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7503

Resources in .sparkStaging directory can't be cleaned up on error

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: YARN
    • Labels:
      None
    • Target Version/s:

      Description

      When we run applications on YARN with cluster mode, uploaded resources on .sparkStaging directory can't be cleaned up in case of failure of uploading local resources.

      You can see this issue by running following command.

      bin/spark-submit --master yarn --deploy-mode cluster --class <someClassName> <non-existing-jar>
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sarutak Kousuke Saruta
                Reporter:
                sarutak Kousuke Saruta
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: