[HDFS-3359] DFSClient.close should close cached sockets

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.22.0, 2.0.0-alpha
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: hdfs-client
• Target Version/s: 2.0.0-alpha
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 03/May/12 09:43
• Updated: 02/Feb/15 19:55
• Resolved: 03/May/12 21:33

Description

Some applications like the TT/JT (pre-2.0) and probably the RM/NM cycle through DistributedFileSystem objects reasonably frequently. So long as they call close() it isn't a big problem, except that currently DFSClient.close() doesn't explicitly close the SocketCache. So unless a full GC runs (causing the references to get finalized), many SocketCaches can get orphaned, each with many open sockets inside. We should fix the close() function to close all cached sockets.

Issue Links

• relates to: HDFS-3373 FileContext HDFS implementation can leak socket caches
• relates to: HDFS-7718 Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext
• relates to: HDFS-941 Datanode xceiver protocol should allow reuse of a connection