[HDFS-5671] Fix socket leak in DFSInputStream#getBlockReader

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: hdfs-client

Environment

hadoop-2.2.0
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
Linux 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

• Hadoop Flags: Reviewed
• Assignee: JamesLi
• Reporter: JamesLi
• Created: 16/Dec/13 10:09
• Updated: 03/Sep/14 23:34
• Resolved: 31/Dec/13 16:44

Description

lsof -i TCP:1004 | grep -c CLOSE_WAIT
18235
When client request a file's block to DataNode:1004. If request fail because "java.io.IOException: Got error for OP_READ_BLOCK,Block token is expired." Occurs and the TCP socket that regionserver using is not closed.

I think the problem above is in DatanodeInfo blockSeekTo(long target) of Class DFSInputStream
The connection client using is BlockReader:
blockReader = getBlockReader(targetAddr, chosenNode, src, blk,
accessToken, offsetIntoBlock, blk.getNumBytes() - offsetIntoBlock,
buffersize, verifyChecksum, dfsClient.clientName);

In DFSInputStream.blockSeekTo()-line 533,invoke getBlockReader() which wil generate a peer use newTcpPeer(dnAddr) -line 1107,when BlockReaderFactory.newBlockReader throw IOException,the peer will not be closed which will cause a CLOSE_WAIT connection.
In our test,when datanode get a InvalidToken exception in DataXceiver.checkAccess(),it will close the connection.At regionserver side, in RemoteBlockReader2.newBlockReader(),checkSuccess() will throw a InvalidBlockTokenException, DFSInputStream.blockSeekTo() will catch the exception, but the connection is NOT closed, it become CLOSE_WAIT.

Issue Links

• is duplicated by: HDFS-5697 connection leak in DFSInputStream