[NIFI-5946] Hive3Streaming memory leak

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.8.0
• Fix Version/s: 1.9.0
• Component/s: Extensions
• Environment: Rhel 7.5 ; Open jdk 1.8 ; NiFi 1.8 ; HDP 3
• Assignee: Unassigned
• Reporter: Gideon Korir
• Created: 10/Jan/19 13:34
• Updated: 10/Jan/19 13:59
• Resolved: 10/Jan/19 13:59

Description

Our NiFi instance has been leaking memory since started using Hive3Streaming processor and now we are seeing memory usage of 60+ GB. Looking at the processor code it seems like we are:

1. Creating a new StreamingConnection with each flow file
2. Registering a shutdown hook via <code>ShutdownHookManager.addShutdownHook</code> that should close the connection
3. Closing the connection and not removing the shutdown hook.

The Runnable captures the connection and conf objects, and since our NiFi instance has been running continuously and the JVM hasn't shutdown (3 weeks +) the ShutdownHookManager is still holding reference to the connections created since causing the memory leak.

Issue Links

• duplicates: NIFI-5841 PutHive3Streaming processor Mem Leak