[HDFS-5099] Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1-beta, 3.0.0-alpha1
• Fix Version/s: 2.1.1-beta
• Component/s: namenode
• Target Version/s: 2.1.1-beta
• Hadoop Flags: Reviewed
• Assignee: Chuan Liu
• Reporter: Chuan Liu
• Created: 15/Aug/13 01:47
• Updated: 12/May/16 18:14
• Resolved: 15/Aug/13 20:55

Description

In Namenode#copyEditLogSegmentsToSharedDir() method, we open a collection of EditLogInputStreams to read and apply to shareEditlog. In readOpt() method, we will open the underlying log file on disk. After applying all the opts, we do not close the collection of streams currently. This lead to a file handle leak on Windows as later we would fail to delete those files.

This happens in TestInitializeSharedEdits test case, where we explicitly called Namenode# initializeSharedEdits(), where copyEditLogSegmentsToSharedDir() is used. Later we fail to create new MiniDFSCluster with the following exception.

java.io.IOException: Could not fully delete C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1
	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
	at org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
…

Issue Links

• is part of: HDFS-3602 Enhancements to HDFS for Windows Server and Windows Azure development and runtime environments