[HIVE-28335] Review deleteOnExitUsage

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: László Bodor
• Created: 18/Jun/24 04:37
• Updated: 16/Aug/24 12:23

Description

This is the followup after HIVE-27884 where I finally decided to leave deleteOnExit as is because I didn't need to change it.

so in the scope of this we need to check and remove all deleteOnExit calls that belong to hadoop FileSystem objects (doesn't necessarily apply to java.io.File.deleteOnExit calls):

grep -iRH "deleteOnExit" --include="*.java" | grep -v "test"
...
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:        // in recent hadoop versions, use deleteOnExit to clean tmp files.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:        autoDelete = fs.deleteOnExit(fsp.outPaths[filesIdx]);
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/PathInfo.java:        fileSystem.deleteOnExit(dir);
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java:      parentDir.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java:      tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/KeyValueContainer.java:        parentDir.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/KeyValueContainer.java:        tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/ObjectContainer.java:        tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java:        autoDelete = fs.deleteOnExit(outPath);

as a reference from previous ticket: commit
it caused some hdfs counters difference in q.outs, needs to investigate

mvn install -DfailIfNoTests=true -Dtest.output.overwrite=true -Pitests,iceberg -Denforcer.skip=true -pl itests/qtest -Dtest=TestMiniLlapCliDriver -Dqfile=orc_llap_counters.q,orc_llap_counters1.q,orc_ppd_basic.q,orc_ppd_schema_evol_3a.q -nsu

Issue Links

• is related to: HIVE-27884 LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon