[HDFS-14292] Introduce Java ExecutorService to DataXceiverServer

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: datanode
• Assignee: David Mollitor
• Reporter: David Mollitor
• Created: 18/Feb/19 14:29
• Updated: 08/Sep/19 12:40

Description

I wanted to investigate dfs.datanode.max.transfer.threads from hdfs-site.xml. It is described as "Specifies the maximum number of threads to use for transferring data in and out of the DN." The default value is 4096. I found it interesting because 4096 threads sounds like a lot to me. I'm not sure how a system with 8-16 cores would react to this large a thread count. Intuitively, I would say that the overhead of context switching would be immense.

During my investigation, I discovered the following setup in the DataXceiverServer class:

1. A peer connects to a DataNode
2. A new thread is spun up to service this connection
3. The thread runs to completion
4. The tread dies

It would perhaps be better if we used a thread pool to better manage the lifecycle of the service threads and to allow the DataNode to re-use existing threads, saving on the need to create and spin-up threads on demand.

In this JIRA, I have added a couple of things:

1. Added a thread pool to DataXceiverServer class that, on demand, will create up to dfs.datanode.max.transfer.threads. A thread that has completed its prior duties will stay idle for up to 60 seconds (configurable), it will be retired if no new work has arrived.
2. Added new methods to the Peer Interface to allow for better logging and less code within each Thread (DataXceiver).
3. Updated the Thread code (DataXceiver) regarding its interactions with blockReceiver instance variable

Issue Links

• relates to: HDFS-12288 Fix DataNode's xceiver count calculation
• relates to: HDFS-14295 Add Threadpool for DataTransfers
• links to: GitHub Pull Request #495