[NIFI-5841] PutHive3Streaming processor Mem Leak

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0, 1.7.1
• Fix Version/s: 1.9.0
• Component/s: Extensions
• Environment: Hive 3.1.*
• Assignee: Kei Miyauchi
• Reporter: Advith Nagappa
• Created: 26/Nov/18 14:46
• Updated: 11/Oct/19 08:04
• Resolved: 09/Jan/19 03:08

Description

Nifi versions: 1,7.1 and 1.8.0
nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/PutHive3Streaming.java**
, line 417 seems redundant,

ShutdownHookManager.addShutdownHook(hiveStreamingConnection::close, FileSystem.SHUTDOWN_HOOK_PRIORITY + 1)

 

Whereas Hive 3.0.0 did not add a shutdownhook within connect() method, Hive 3.1.* does:

 hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java

ShutdownHookManager.addShutdownHook(streamingConnection::close, FileSystem.SHUTDOWN_HOOK_PRIORITY + 1);

This creates two references to the shutdownhook object per transaction out of which only one is ever cleaned; resulting in a slow/fast degradation of heap space depending on the velocity of transactions.

Issue Links

• is duplicated by: NIFI-5946 Hive3Streaming memory leak
• is related to: HIVE-20979 Fix memory leak in hive streaming
• links to: GitHub Pull Request #3249