Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5352

File handle cache needs timeout based eviction

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:
      None

      Description

      The file handle cache currently will keep file handles open indefinitely if the cache is not at its maximum capacity. This means that file handles might stay around for extended periods of time (weeks, months). Since local files are accessed directly, an open file handle can prevent the disk blocks from being freed, even if the file is deleted through HDFS. The file handle cache should implement a timeout for file handles so that a file handle that is not used recently will be evicted. This limit should be configurable and it may be desirable for the default to take into account HDFS's fs.trash.interval.

      Additionally, when files are replaced or appended, the file's mtime will increase. File handles with the old mtime will no longer be accessed, but they may not be aged out of the cache. These should be aged out more aggressively.

        Attachments

          Activity

            People

            • Assignee:
              joemcdonnell Joe McDonnell
              Reporter:
              joemcdonnell Joe McDonnell
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: