Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30069

Clean up non-shuffle disk block manager files following executor exists on YARN

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Spark Core, YARN
    • None

    Description

      Currently we only clean up the local directories on application removed. However, when executors die and restart repeatedly, many temp files are left untouched in the local directories, which is undesired behavior and could cause disk space used up gradually.
      SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this issue still exists. Especially, in long running service like Spark thrift-server with dynamic resource allocation disabled, it's very easy causes local disk full.

      Attachments

        Activity

          People

            Unassigned Unassigned
            cltlfcjin Lantao Jin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: