[HDFS-3672] Expose disk-location information for blocks to enable better scheduling

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.0.2-alpha
• Hadoop Flags: Reviewed
• Assignee: Andrew Wang
• Reporter: Andrew Wang
• Created: 16/Jul/12 22:05
• Updated: 09/Oct/13 22:45
• Resolved: 17/Aug/12 18:04

Description

Currently, HDFS exposes on which datanodes a block resides, which allows clients to make scheduling decisions for locality and load balancing. Extending this to also expose on which disk on a datanode a block resides would enable even better scheduling, on a per-disk rather than coarse per-datanode basis.

This API would likely look similar to Filesystem#getFileBlockLocations, but also involve a series of RPCs to the responsible datanodes to determine disk ids.

Issue Links

• is depended upon by: HBASE-6572 Tiered HFile storage
• is related to: HDFS-3969 Small bug fixes and improvements for disk locations API
• is related to: MAPREDUCE-4577 HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
• is related to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
• is related to: HBASE-6572 Tiered HFile storage