Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31346

Add new configuration to make sure temporary directory cleaned

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to keep deleteOnExit, even if temporary directory has already deleted, to make sure the temporary directories cleaned.

      Attachments

        Issue Links

          Blocked

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-31438 Support JobCleaned Status in SparkListener

          • Major - Major loss of function.
          • In Progress
          links to

          Web Link GitHub Pull Request #28129

          Activity

            People

              Unassigned Unassigned
              Jackey Lee Jackey Lee
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: