[YARN-5939] FSDownload leaks FileSystem resources

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.5.1, 2.7.3
• Component/s: nodemanager
• Labels: leak
• Flags: Important
• Assignee: Weiwei Yang
• Reporter: liuxiangwei
• Created: 29/Nov/16 09:05
• Updated: 28/Nov/19 07:18
• Resolved: 26/Sep/18 10:47

Description

Background
To use our self-defined FileSystem class, the item of configuration "fs.%s.impl.disable.cache" should set to true.

In YARN's source code, the class named "org.apache.hadoop.yarn.util.FSDownload" use getFileSystem but never close, which leading to file descriptor leak because our self-defined FileSystem class close the file descriptor when the close function is invoked.

My Question below:
1. whether invoking "getFileSystem" but never close is YARN's expected behavior
2. what should we do in our self-defined FileSystem resolve it.

Issue Links

• is duplicated by: YARN-58 NM leaks filesystems