[HDFS-5682] Heterogeneous Storage phase 2 - APIs to expose Storage Types

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 2.7.0
• Component/s: datanode, namenode
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 18/Dec/13 20:19
• Updated: 30/Jun/15 07:25
• Resolved: 13/May/15 20:48

Description

Phase 1 (HDFS-2832) added support to present the DataNode as a collection of discrete storages of different types.

This Jira is to track phase 2 of the Heterogeneous Storage work which involves exposing Storage Types to applications and adding Quota Management support for administrators.

This phase will also include tools support for administrators/users.

Issue Links

• is related to: HDFS-5851 Support memory as a storage medium
• is related to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
• is related to: HDFS-6584 Support Archival Storage
• relates to: HBASE-6572 Tiered HFile storage

Sub-Tasks

• Namenode webUI changes to incorporate storage type: Open
• writeBlock should accept a target storage directory: Resolved
• Add a storage type attribute to file: Resolved
• Balancer needs to consider storage type in balancing: Resolved
• DataTransferProtocol changes for supporting mutiple storages per DN: Resolved
• Consider renaming StorageID: Resolved
• Verify initializations of LocatedBlock/RecoveringBlock: Resolved
• Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a global view in choosing DN storage for replica.: Resolved
• Add test case for commitBlockSynchronization with invalid Storage: Open
• StorageType should be encoded in the block token: Resolved