[HDFS-2246] Shortcut a local client reads to a Datanodes files directly

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.1, 1.0.0, 0.22.1
• Target Version/s: 0.23.1, 1.0.0
• Hadoop Flags: Reviewed

Release Note

1. New configurations
a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
2. By default none of the above are enabled and short circuit read will not kick in.
3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.

• Assignee: Jitendra Nath Pandey
• Reporter: Sanjay Radia
• Created: 11/Aug/11 17:35
• Updated: 17/Jul/14 18:54
• Resolved: 26/Nov/11 11:00

Issue Links

• duplicates: HDFS-281 Explore usage of the sendfile api via java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes
• is related to: HADOOP-7804 enable hadoop config generator to set dfs.block.local-path-access.user to enable short circuit read
• is related to: HDFS-3907 Allow multiple users for local block readers
• relates to: HADOOP-6311 Add support for unix domain sockets to JNI libs
• relates to: HDFS-245 Create symbolic links in HDFS
• relates to: HDFS-347 DFS read performance suboptimal when client co-located on nodes with data
• relates to: HDFS-2654 Make BlockReaderLocal not extend RemoteBlockReader2