[HDFS-347] DFS read performance suboptimal when client co-located on nodes with data

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: datanode, hdfs-client, performance
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Colin McCabe
• Reporter: George Porter
• Created: 07/Dec/08 19:34
• Updated: 30/Nov/16 03:30
• Resolved: 16/May/13 07:11

Description

One of the major strategies Hadoop uses to get scalable data processing is to move the code to the data. However, putting the DFS client on the same physical node as the data blocks it acts on doesn't improve read performance as much as expected.

After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem is due to the HDFS streaming protocol causing many more read I/O operations (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB disk block from the DataNode process (running in a separate JVM) running on the same machine. The DataNode will satisfy the single disk block request by sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, this is done in the sendChunk() method, relying on Java's transferTo() method. Depending on the host O/S and JVM implementation, transferTo() is implemented as either a sendfilev() syscall or a pair of mmap() and write(). In either case, each chunk is read from the disk by issuing a separahitting te I/O operation for each chunk. The result is that the single request for a 64-MB block ends up the disk as over a thousand smaller requests for 64-KB each.

Since the DFSClient runs in a different JVM and process than the DataNode, shuttling data from the disk to the DFSClient also results in context switches each time network packets get sent (in this case, the 64-kb chunk turns into a large number of 1500 byte packet send operations). Thus we see a large number of context switches for each block send operation.

I'd like to get some feedback on the best way to address this, but I think providing a mechanism for a DFSClient to directly open data blocks that happen to be on the same machine. It could do this by examining the set of LocatedBlocks returned by the NameNode, marking those that should be resident on the local host. Since the DataNode and DFSClient (probably) share the same hadoop configuration, the DFSClient should be able to find the files holding the block data, and it could directly open them and send data back to the client. This would avoid the context switches imposed by the network layer, and would allow for much larger read buffers than 64KB, which should reduce the number of iops imposed by each read block operation.

Issue Links

• blocks: HBASE-8337 Investigate why disabling hadoop short circuit read is required to make recovery tests pass consistently under hadoop2
• contains: HADOOP-9983 SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not public class in 2.0.5-alpha
• depends upon: HADOOP-6311 Add support for unix domain sockets to JNI libs
• is depended upon by: ACCUMULO-884 Insight into short circuit read for local files
• is related to: HDFS-1599 Umbrella Jira for Improving HBASE support in HDFS
• is related to: HDFS-6699 Secure Windows DFS read when client co-located on nodes with data (short-circuit reads)
• is related to: HBASE-3529 Add search to HBase
• is related to: HDFS-2246 Shortcut a local client reads to a Datanodes files directly
• relates to: HDFS-4284 BlockReaderLocal not notified of failed disks
• relates to: HADOOP-3205 Read multiple chunks directly from FSInputChecker subclass into user buffers
• relates to: HDFS-385 Design a pluggable interface to place replicas of blocks in HDFS

Sub-Tasks

• Encapsulate arguments to BlockReaderFactory in a class: Resolved
• Encapsulate connections to peers in Peer and PeerServer classes: Resolved
• Create DomainSocket and DomainPeer and associated unit tests: Resolved
• BlockReaderLocal should use passed file descriptors rather than paths: Resolved
• DomainSocket should throw AsynchronousCloseException when appropriate: Resolved
• Bypass UNIX domain socket unit tests when they cannot be run: Resolved
• DFSInputStream#getBlockReader: last retries should ignore the cache: Resolved
• Fix bug in DomainSocket path validation: Resolved
• some small DomainSocket fixes: avoid findbugs warning, change log level, etc.: Resolved
• change dfs.datanode.domain.socket.path to dfs.domain.socket.path: Resolved
• HDFS-347: fix case where local reads get disabled incorrectly: Resolved
• HDFS-347: increase default FileInputStreamCache size: Resolved
• make TestPeerCache not flaky: Resolved
• TestDomainSocket fails when system umask is set to 0002: Resolved
• avoid annoying log message when dfs.domain.socket.path is not set: Resolved
• Make a simple doc to describe the usage and design of the shortcircuit read feature: Resolved
• DataNode: don't create domain socket unless we need it: Resolved
• HDFS-347: style cleanups: Resolved
• HDFS-347: DN should chmod socket path a+w: Closed
• DFSClient: don't create a domain socket unless we need it: Resolved
• allow use of legacy blockreader: Resolved
• fix various bugs in short circuit read: Closed