[HADOOP-15658] Memory leak in S3AOutputStream

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.4
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Piotr Nowojski
• Created: 09/Aug/18 12:01
• Updated: 20/Jun/19 10:58
• Resolved: 20/Jun/19 10:58

Description

S3AOutputStream by calling org.apache.hadoop.fs.s3a.S3AFileSystem#createTmpFileForWrite indirectly calls java.io.File#deleteOnExit and java.io.DeleteOnExitHook which are known for memory leaking:
 
https://bugs.java.com/view_bug.do?bug_id=6664633
https://bugs.java.com/view_bug.do?bug_id=4872014
 
Apparently it was even fixed (same bug but in unrelated issue) for different component couple of years ago https://issues.apache.org/jira/browse/HADOOP-8635 . 

Issue Links

• Is contained by: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename