Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31346

Add new configuration to make sure temporary directory cleaned

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to keep deleteOnExit, even if temporary directory has already deleted, to make sure the temporary directories cleaned.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jackey Lee Jackey Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: