[HDFS-5484] StorageType and State in DatanodeStorageInfo in NameNode is not accurate

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Heterogeneous Storage (HDFS-2832)
• Fix Version/s: Heterogeneous Storage (HDFS-2832)
• Component/s: datanode
• Target Version/s: Heterogeneous Storage (HDFS-2832)
• Hadoop Flags: Reviewed
• Assignee: Eric Sirianni
• Reporter: Eric Sirianni
• Created: 08/Nov/13 18:45
• Updated: 14/Dec/13 03:57
• Resolved: 03/Dec/13 16:33

Description

The fields in DatanodeStorageInfo are updated from two distinct paths:

1. block reports
2. storage reports (via heartbeats)

The state and storageType fields are updated via the Block Report. However, as seen in the code blow, these fields are populated from a "dummy" DatanodeStorage object constructed in the DataNode:

BPServiceActor.blockReport() {
//...
        // Dummy DatanodeStorage object just for sending the block report.
        DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}

The net effect is that the state and storageType fields are always the default of NORMAL and DISK in the NameNode.

The recommended fix is to change FsDatasetSpi.getBlockReports() from:

public Map<String, BlockListAsLongs> getBlockReports(String bpid);

to:

public Map<DatanodeStorage, BlockListAsLongs> getBlockReports(String bpid);

thereby allowing BPServiceActor to send the "real" DatanodeStorage object with the block report.

Issue Links

• is cloned by: HDFS-5667 Include DatanodeStorage in StorageReport