Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24340

Clean up non-shuffle disk block manager files following executor death

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Currently we only clean up local folders on application removed, and we don't clean up non-shuffle files, such as temp. shuffle blocks, cached RDD/broadcast blocks, spill files, etc. and this can cause disk space leaks when executors periodically die and are replaced.

      To avoid this source of disk space leak, we can clean up executor disk store files except for shuffle index and data files on executor finished.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jiangxb1987 Xingbo Jiang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: