[HDFS-5182] BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: hdfs-client
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 11/Sep/13 00:03
• Updated: 12/May/16 18:16
• Resolved: 08/Sep/14 20:03

Description

BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid. This implies adding a new field to the response to REQUEST_SHORT_CIRCUIT_FDS. We also need some kind of heartbeat from the client to the DN, so that the DN can inform the client when the mapped region is no longer locked into memory.

Issue Links

• is blocked by: HDFS-5634 allow BlockReaderLocal to switch between checksumming and not
• relates to: HDFS-5521 skip checksums when reading a cached block via non-local reads
• relates to: HDFS-5394 fix race conditions in DN caching and uncaching

Sub-Tasks

• add ShortCircuitSharedMemorySegment: Closed
• Unify mmap cache and short-circuit file descriptor cache: Closed
• Make BlockReaderFactory#BlockReaderPeer a static class: Closed
• Minor cleanups to ShortCircuitReplica, FsDatasetCache, and DomainSocketWatcher: Closed
• add DomainSocket#shutdown method: Closed
• The DFSClient and DataNode should use shared memory segments to communicate short-circuit information: Closed
• fix DFSClient issue without libhadoop.so and some other ShortCircuitShm cleanups: Closed
• add dfs.client.mmap.enabled: Closed
• Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed: Closed
• Fix a case where zero-copy or no-checksum reads were not allowed even when the block was cached: Closed
• The DataNode should use its shared memory segment to mark short-circuit replicas that have been unlinked as stale: Closed
• Forcibly timeout misbehaving DFSClients that try to do no-checksum reads that extend too long: Closed