Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27014

Support removal of jars and Spark binaries from Mesos driver and executor sandboxes

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Mesos
    • None

    Description

      Currently, each Spark application run on Mesos leaves behind at least 500MB of data in sandbox directories, coming from Spark binaries and copied URIs. These can build up as a disk leak, causing major issues on Mesos clusters unless their grace period for sandbox directories is very short.

      Spark should have a feature to delete these (from both driver and executor sandboxes) on teardown.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mwlon Martin Loncaric
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: