Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7705

Cleanup of .sparkStaging directory fails if application is killed

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.4.1, 1.5.0
    • Component/s: YARN
    • Labels:
      None

      Description

      When a streaming application is killed while running on YARN the .sparkStaging directory is not cleaned up. Setting spark.yarn.preserve.staging.files=false does not help and still leaves the files around.

      The changes in SPARK-7503 do not catch this case since there is no exception in the shutdown. When the application gets killed the AM gets told to shutdown and the shutdown hook is run but the clean up is not triggered.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Sephiroth-Lin Weizhong
                Reporter:
                wilfreds Wilfred Spiegelenburg
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: