Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.9.0
-
None
-
ghx-label-6
Description
The file handle cache currently will keep file handles open indefinitely if the cache is not at its maximum capacity. This means that file handles might stay around for extended periods of time (weeks, months). Since local files are accessed directly, an open file handle can prevent the disk blocks from being freed, even if the file is deleted through HDFS. The file handle cache should implement a timeout for file handles so that a file handle that is not used recently will be evicted. This limit should be configurable and it may be desirable for the default to take into account HDFS's fs.trash.interval.
Additionally, when files are replaced or appended, the file's mtime will increase. File handles with the old mtime will no longer be accessed, but they may not be aged out of the cache. These should be aged out more aggressively.