[HDFS-4161] HDFS keeps a thread open for every file writer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.1
• Component/s: hdfs-client
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Suresh Srinivas
• Created: 07/Nov/12 22:14
• Updated: 04/Dec/12 19:01
• Resolved: 09/Nov/12 20:53

Description

In 1.0 release DFSClient uses a thread per file writer. In some use cases (dynamic partions in hive) that use a large number of file writers a large number of threads are created. The file writer thread has the following stack:

at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1462)
at java.lang.Thread.run(Thread.java:662)

This problem has been fixed in later releases. This jira will post a consolidated patch from various jiras that addresses the issue.

Issue Links

• relates to: HADOOP-9115 Deadlock in configuration when writing configuration to hdfs
• relates to: HDFS-1865 Share LeaseChecker thread among DFSClients