Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.3.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The fix for HDFS-5484 was accidentally regressed by the following change made via HDFS-5542

      +  DatanodeStorageInfo updateStorage(DatanodeStorage s) {
           synchronized (storageMap) {
             DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
             if (storage == null) {
      @@ -670,8 +658,6 @@
                        " for DN " + getXferAddr());
               storage = new DatanodeStorageInfo(this, s);
               storageMap.put(s.getStorageID(), storage);
      -      } else {
      -        storage.setState(s.getState());
             }
             return storage;
           }
      

      By removing the 'else' and no longer updating the state in the BlockReport processing path, we effectively get the bogus state & type that is set via the first heartbeat (see the fix for HDFS-5455):

      +      if (storage == null) {
      +        // This is seen during cluster initialization when the heartbeat
      +        // is received before the initial block reports from each storage.
      +        storage = updateStorage(new DatanodeStorage(report.getStorageID()));
      

      Even reverting the change and reintroducing the 'else' leaves the state & type temporarily inaccurate until the first block report.

      As discussed with [~arpitagarwal], a better fix would be to simply include the full DatanodeStorage object in the StorageReport (as opposed to only the Storage ID). This requires adding the DatanodeStorage object to StorageReportProto. It needs to be a new optional field and we cannot remove the existing StorageUuid for protocol compatibility.

        Attachments

        1. h5667.05.patch
          31 kB
          Arpit Agarwal
        2. h5667.04.patch
          32 kB
          Arpit Agarwal
        3. h5667.03.patch
          31 kB
          Arpit Agarwal
        4. h5667.02.patch
          31 kB
          Arpit Agarwal

          Issue Links

            Activity

              People

              • Assignee:
                arp Arpit Agarwal
                Reporter:
                sirianni Eric Sirianni
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: