[HDFS-916] Rewrite DFSOutputStream to use a single thread with NIO

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.22.0
• Component/s: hdfs-client
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 23/Jan/10 00:35
• Updated: 12/Aug/19 23:12

Description

The DFS write pipeline code has some really hairy multi-threaded synchronization. There have been a lot of bugs produced by this (HDFS-101, HDFS-793, HDFS-915, tens of others) since it's very hard to understand the message passing, lock sharing, and interruption properties. The reason for the multiple threads is to be able to simultaneously send and receive. If instead of using multiple threads, it used nonblocking IO, I think the whole thing would be a lot less error prone.

I think we could do this in two halves: one half is the DFSOutputStream. The other half is BlockReceiver. I opened this JIRA first as I think it's simpler (only one TCP connection to deal with, rather than an up and downstream)

Opinions? Am I crazy? I would like to see some agreement on the idea before I spend time writing code.

Issue Links

• is related to: HDFS-223 Asynchronous IO Handling in Hadoop and HDFS
• is related to: AVRO-405 Netty-based Java RPC server
• is related to: HDFS-281 Explore usage of the sendfile api via java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes
• is related to: MAPREDUCE-258 Update MapOutputServlet to use NIO channels
• is related to: MAPREDUCE-319 Use Grizzly for Fetching Map Output in Shuffle
• is related to: HADOOP-3518 Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem
• relates to: HBASE-14790 Implement a new DFSOutputStream for logging WAL only
• relates to: HBASE-2182 rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.
• requires: HDFS-914 Refactor DFSOutputStream and DFSInputStream out of DFSClient
• links to: Asynchronous DFS Output Proposal