[HIVE-27884] LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 17/Nov/23 18:47
• Updated: 03/Feb/24 00:19

Description

Originally, when the task runner was added to HIVE-10028 (here), the FileSystem.closeAllForUGI was commented out for some reasons, and then, in the scope of HIVE-9898 it was simply added back, here

A FileSystem.close call basically does the following:
1. delete all paths that were marked as delete-on-exit.
2. removes the instance from the cache

I saw that we call

FileSystem.closeAllForUGI

at the end of all task attempts, so we almost completely disable hadoop's filesystem cache during a long-running LLAP daemon lifecycle

some investigations on azure showed that creating a filesystem can be quite expensive, as it involves the recreation of a whole object hierarchy like:

AzureBlobFileSystem -> AzureBlobFileSystemStore --> AbfsClient -> TokenProvider(MsiTokenProvider)

which ends up pinging the token auth endpoint of azure, leading to e.g. a HTTP response 429

We need to check whether we can remove this closeAllForUGI in LLAP, additionally check and remove all deleteOnExit calls that belong to hadoop FileSystem objects (doesn't necessarily apply to java.io.File.deleteOnExit calls):

grep -iRH "deleteOnExit" --include="*.java" | grep -v "test"
...
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:        // in recent hadoop versions, use deleteOnExit to clean tmp files.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:        autoDelete = fs.deleteOnExit(fsp.outPaths[filesIdx]);
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/PathInfo.java:        fileSystem.deleteOnExit(dir);
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java:      parentDir.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java:      tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/KeyValueContainer.java:        parentDir.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/KeyValueContainer.java:        tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/ObjectContainer.java:        tmpFile.deleteOnExit();
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java:        autoDelete = fs.deleteOnExit(outPath);

I believe deleteOnExit is fine if we don't want to bother with removing temp files, however, these deletions might want to go to a more hive-specific scope if we want to really reuse cached filesystems in a safe manner.

Issue Links

• is related to: HIVE-9898 LLAP: Sort out issues with UGI and cached FileSystems
• is related to: HIVE-10028 LLAP: Create a fixed size execution queue for daemons
• is related to: HADOOP-17377 ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
• is related to: HIVE-131 insert overwrite directory leaves behind uncommitted/tmp files from failed tasks
• links to: GitHub Pull Request #4882