[HDFS-5667] Include DatanodeStorage in StorageReport

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.3.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Eric Sirianni
• Created: 13/Dec/13 21:31
• Updated: 12/May/16 18:17
• Resolved: 06/Jan/14 17:28

Description

The fix for HDFS-5484 was accidentally regressed by the following change made via HDFS-5542

+  DatanodeStorageInfo updateStorage(DatanodeStorage s) {
     synchronized (storageMap) {
       DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
       if (storage == null) {
@@ -670,8 +658,6 @@
                  " for DN " + getXferAddr());
         storage = new DatanodeStorageInfo(this, s);
         storageMap.put(s.getStorageID(), storage);
-      } else {
-        storage.setState(s.getState());
       }
       return storage;
     }

By removing the 'else' and no longer updating the state in the BlockReport processing path, we effectively get the bogus state & type that is set via the first heartbeat (see the fix for HDFS-5455):

+      if (storage == null) {
+        // This is seen during cluster initialization when the heartbeat
+        // is received before the initial block reports from each storage.
+        storage = updateStorage(new DatanodeStorage(report.getStorageID()));

Even reverting the change and reintroducing the 'else' leaves the state & type temporarily inaccurate until the first block report.

As discussed with arpitagarwal, a better fix would be to simply include the full DatanodeStorage object in the StorageReport (as opposed to only the Storage ID). This requires adding the DatanodeStorage object to StorageReportProto. It needs to be a new optional field and we cannot remove the existing StorageUuid for protocol compatibility.

Issue Links

• is a clone of: HDFS-5484 StorageType and State in DatanodeStorageInfo in NameNode is not accurate