Details
-
Improvement
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
0.23.9, 0.23.10, 2.2.0, 2.3.0, 3.0.0-alpha1
-
None
-
-
Improvement of DFSClient deadNodes list handling, which speeds up dead datanode discovery. DFSClient dfsInputStream threads can now share this information among them, improving overall HDFS performance.
-
DFSClient, deadNodes
Description
This patch solves an issue of deadNodes list being thread local. deadNodes list is created by DFSClient when some problems with write/reading, or contacting datanode exist. The problem is that deadNodes is not visible to other DFSInputStream threads, hence every DFSInputStream ends up building its own deadNodes. This affect performance of DFSClient to a large degree especially when a datanode goes completely offline (there is a tcp connect delay experienced by all DFSInputStream threads affecting performance of the whole cluster).
This patch moves deadNodes to be global in DFSClient class so that as soon as a single DFSInputStream thread reports a dead datanode, all other DFSInputStream threads are informed, negating the need to create their own independent lists (concurrent Map really).
Further, a global deadNodes health check manager thread (DeadNodeVerifier) is created to verify all dead datanodes every 5 seconds, and remove the same list as soon as it is up. That thread under normal conditions (deadNodes empty) would be sleeping. If deadNodes is not empty, the thread will attempt to open tcp connection every 5 seconds to affected datanodes.
This patch has a test (TestDFSClientDeadNodes) that is quite simple, since the deadNodes creation is not affected by the patch, we only test datanode removal from deadNodes by the health check manager thread. Test will create a file in dfs minicluster, read from the same file rapidly, cause datanode to restart, and test is the health check manager thread does the right thing, removing the alive datanode from the global deadNodes list.
