Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7503

Resources in .sparkStaging directory can't be cleaned up on error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.0
    • 1.4.0
    • YARN
    • None

    Description

      When we run applications on YARN with cluster mode, uploaded resources on .sparkStaging directory can't be cleaned up in case of failure of uploading local resources.

      You can see this issue by running following command.

      bin/spark-submit --master yarn --deploy-mode cluster --class <someClassName> <non-existing-jar>

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-7439 Should delete temporary local directories

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7705 Cleanup of .sparkStaging directory fails if application is killed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #6026 [Github] Pull Request #6026 (sarutak)

          Activity

            People

              sarutak Kousuke Saruta
              sarutak Kousuke Saruta
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: