[NIFI-6790] Load-balanced queues cause huge number of open pipes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.2
• Fix Version/s: 1.11.3
• Component/s: Core Framework
• Labels: Cluster LoadBalancing
• Environment: Ubuntu,OpenJDK8
• Assignee: Unassigned
• Reporter: Matthew Knight
• Created: 17/Oct/19 19:58
• Updated: 14/Mar/22 21:05
• Resolved: 14/Mar/22 21:05

Description

Even very basic flows using load balance connecting queues result in huge numbers of open pipes which don't seem to get closed and result in the entire cluster being brought to a standstill because of "too many open files".  I tried apples-to-apples with equivalent very basic flows, one load balanced and one not, to show how easy this has been to reproduce.  I'm attaching some screenshots that show the "too many open files" issue in action in my NiFi cluster, along with the output of LSOF for the primary node of a cluster with vs. without load balancing.  When load balancing is enabled there is a huge number of 'pipe' and 'eventpoll' items, without load balancing things are a bit more balanced.

Issue Links

• is fixed by: NIFI-7114 DistributedMapCacheClient/Server not closing resources leading to exhausted file handles
• relates to: NIFI-7114 DistributedMapCacheClient/Server not closing resources leading to exhausted file handles