[HIVE-20979] Fix memory leak in hive streaming

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.1.1, 4.0.0-alpha-1
• Component/s: Streaming
• Labels: pull-request-available
• Assignee: Shubham Chaurasia
• Reporter: Shubham Chaurasia
• Created: 28/Nov/18 08:06
• Updated: 17/Nov/22 08:54
• Resolved: 10/Dec/18 09:40

Description

1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via ShutdownHookManager.addShutdownHook but it is never removed which causes all the handlers to accumulate and hence a memory leak.

2) AbstractRecordWriter creates an instance of FileSystem but does not close it which in turn causes a leak due to accumulation in FileSystem$Cache#map

 

Issue Links

• relates to: NIFI-5841 PutHive3Streaming processor Mem Leak
• links to: GitHub Pull Request #495
• links to: GitHub Pull Request #495