[HDFS-4990] Block placement for storage types

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Heterogeneous Storage (HDFS-2832)
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Suresh Srinivas
• Created: 12/Jul/13 22:36
• Updated: 18/Sep/13 22:20
• Resolved: 18/Sep/13 15:13

Description

Currently block location for writes are made based on:

1. Datanode load (number of transceivers)
2. Space left on datanode
3. Topology

With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc.

As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well.

Issue Links

• is depended upon by: HDFS-4989 Balancer needs to consider storage type in balancing