[HDFS-4953] enable HDFS local reads via mmap

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: HDFS-4949
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 03/Jul/13 01:10
• Updated: 27/Jun/20 05:28
• Resolved: 20/Aug/13 18:17

Description

Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

Issue Links

• blocks: HDFS-5101 ZCR should work with blocks bigger than 2 GB
• is duplicated by: HDFS-3053 Support for true zero-copy (mmap-based) reads
• is part of: HDFS-5260 Merge zero-copy memory-mapped HDFS client reads to trunk and branch-2.
• is related to: HDFS-5191 revisit zero-copy API in FSDataInputStream to make it more intuitive
• is related to: HDFS-5109 Document new zero-copy short-circuit local reads feature
• relates to: HDFS-4949 Centralized cache management in HDFS
• relates to: HDFS-6007 Update documentation about short-circuit local reads