[NIFI-7454] Directory handle leak in ListFile

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.11.4
• Fix Version/s: 1.12.0
• Assignee: Simon Bence
• Reporter: Paul Kelly
• Created: 15/May/20 09:47
• Updated: 19/May/20 15:43
• Resolved: 19/May/20 12:43

Description

ListFile does not seem to be closing directory handles if an exception is raised during the directory walk.  For example, if ListFile is recursively polling a folder and the NiFi user does not have permission to list the contents of one of the subfolders, an "Access Denied" bulletin is raised and the number of DIR handles opened by the NiFi process as shown by lsof increases.  Eventually, NiFi stops working due to "Too many open files".

This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11.  I'm not sure in which version this behavior started.

Attached is a sample flow.xml.gz that demonstrates this behavior.  A clean copy of NiFi 1.11.4 was installed with no modifications to any configuration file.  NiFi is running as the user "xubuntu".  The permissions on the folders are as follows:
drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test
drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest

NiFi has permission to list the contents of "/home/xubuntu/test", but not "/home/xubuntu/test/subtest".

The number of open DIR handles is being monitored with the command:
lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c
where 347813 is the pid of the NiFi process.

DIR handle counts start at 4 immediately after NiFi is launched and steadily grow to tens of thousands, until file handles are exhausted.  Additionally, these handles do not seem to be released even if the ListFile processor is stopped.

Please let me know if there is any additional information I can provide.

Issue Links

• is fixed by: NIFI-7292 ListFile should skip elements without sufficient permissions