[HDFS-6581] Write to single replica in memory

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: datanode, hdfs-client, namenode
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 20/Jun/14 18:56
• Updated: 17/Jun/15 19:09
• Resolved: 21/Oct/14 20:28

Description

Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol.

This avoids some of the issues with short-circuit writes, which we can revisit at a later time.

Issue Links

• is related to: HDFS-6912 SharedFileDescriptorFactory should not allocate sparse files
• is required by: HDFS-5851 Support memory as a storage medium
• relates to: HDFS-6919 Enforce a single limit for RAM disk usage and replicas cached via locking
• requires: HDFS-6898 DN must reserve space for a full block when an RBW block is created
• requires: HDFS-6981 Fix DN upgrade with layout version change
• requires: HDFS-6482 Use block ID-based block layout on datanodes
• requires: HDFS-6800 Support Datanode layout changes with rolling upgrade

Sub-Tasks

• Initial prototype implementation for replicas in memory using tmpfs: Resolved
• Add LazyPersist flag to FileStatus: Closed
• The HttpFSFileSystem should also return the isLazyPersist flag in the FileStatus: Resolved
• Add LazyPersist flag to INodeFile, save it in FsImage and edit logs: Resolved
• Propagate LazyPersist flag to DNs via DataTransferProtocol: Closed
• Add new RAM_DISK storage type: Closed
• DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received: Closed
• DN support for saving replicas to persistent storage and evicting in-memory replicas: Closed
• Add unit tests: Closed
• NN periodically unlinks lazy persist files with missing replicas from namespace: Closed
• 'hdfs put' command should accept lazyPersist flag for testing: Closed
• Bugfix in LazyWriter, fix test case and some refactoring: Closed
• Move lazily persisted replicas to finalized directory on DN startup: Closed
• Add Additional unit tests for HDFS-6581: Closed
• Improve replica eviction from RAM disk: Closed
• Delete all copies when a block is deleted from the block space: Closed
• Notify NN of evicted block before deleting it from RAM disk: Closed
• Directory scanner should correctly reconcile blocks on RAM disk: Closed
• LazyWriter#evictBlocks misses a null check for replicaState: Closed
• Fix unit test failures in HDFS-6581 branch: Closed
• Few more unit test fixes for HDFS-6581: Closed
• Fix finalize and upgrade unit test failures: Closed
• FsDatasetImpl#copyBlockFiles debug log can be improved: Closed
• Add forwarding constructor for INodeFile for existing callers: Closed
• Updated editsStored and editsStored.xml to bump layout version and add LazyPersist flag: Resolved
• Fix unit test failures in SimulatedFsDataset: Closed
• Make eviction scheme pluggable: Closed
• Add unit test for evict/delete RAM_DISK block with open handle: Closed
• Fix findbugs warnings in the HDFS-6581 branch: Resolved
• Balancer and Mover tools should ignore replicas on RAM_DISK: Closed
• Fix findbugs warnings in HDFS-6581 branch: Closed
• Fix findbugs warnings in RamDiskReplicaTracker: Closed
• Integrate HDFS-6581 memory usage with HDFS-4949: Resolved
• Bugfix in createLocatedFileStatus caused by bad merge: Closed
• Fix incorrect layout version caused by bad merge: Resolved
• Use block storage policy to set lazy persist preference: Closed
• Metrics to track usage of memory for writes: Closed
• Fix Jenkins failures in HDFS-6581 branch: Closed
• Improve HDFS-6581 eviction configuration: Closed
• Use unbuffered writes when persisting in-memory replicas: Closed
• LazyWriter should use either async IO or one thread per physical disk: Closed
• Move checksum computation off the hot path when writing to RAM disk: Closed
• Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows: Closed
• Feature documentation for HDFS-6581: Closed
• DN should ignore lazyPersist hint if the writer is not local: Resolved
• Implement a 2Q eviction strategy for HDFS-6581: Patch Available
• Skip checksum verification when reading from memory: Open
• Investigate eviction from RAM disk via file truncation: Open
• DFSOutputStream should query policies from NameNode: Open
• Support using ramfs partitions on Linux: Open