[HDFS-11118] Block Storage for HDFS

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs
• Assignee: Anu Engineer
• Reporter: Anu Engineer
• Created: 07/Nov/16 23:56
• Updated: 11/Nov/16 23:30

Description

This JIRA proposes extending HDFS to provide replicated block storage capabilities using Storage Containers. This is would allow users to run unmodified programs that assume that they are running on a posix file system.

With this extension, HDFS can be used like a block store. For example, YARN jobs could mount and use a volume at will. This is made possible by leveraging Storage Containers and will share the storage layer with Ozone and HDFS in future.

Please see the attached design document for more details on this proposal.

Issue Links

• is related to: HDFS-7240 Scaling HDFS

Sub-Tasks

• Block Storage : configurable max cache size: Open
• Add documentation for metrics in CBlockTargetMetrics to OzoneMetrics.md: Open
• Add markdown documentation about Cblock: Open
• Block Storage: Fix cblock unit test failures in TestLocalBlockCache, TestCBlockReadWrite and TestBufferManager: Open
• Block Storage: Add flushid in place of timestamp in DirtyLog file signatures: Open
• Block Storage: Add general abstraction for metadata store in CBlock: Open
• Block Storage: Add start-cblock.sh to quickly start cblock.: Open