[HDFS-6581] Write to single replica in memory

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: datanode, hdfs-client, namenode
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 20/Jun/14 18:56
• Updated: 17/Jun/15 19:09
• Resolved: 21/Oct/14 20:28

Description

Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol.

This avoids some of the issues with short-circuit writes, which we can revisit at a later time.

Issue Links

• is related to: HDFS-6912 SharedFileDescriptorFactory should not allocate sparse files
• is required by: HDFS-5851 Support memory as a storage medium
• relates to: HDFS-6919 Enforce a single limit for RAM disk usage and replicas cached via locking
• requires: HDFS-6898 DN must reserve space for a full block when an RBW block is created
• requires: HDFS-6981 Fix DN upgrade with layout version change
• requires: HDFS-6482 Use block ID-based block layout on datanodes
• requires: HDFS-6800 Support Datanode layout changes with rolling upgrade

Sub-Tasks

• Implement a 2Q eviction strategy for HDFS-6581: Patch Available
• Skip checksum verification when reading from memory: Open
• Investigate eviction from RAM disk via file truncation: Open
• DFSOutputStream should query policies from NameNode: Open
• Support using ramfs partitions on Linux: Open