Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5505

DataStorage.format() should ignore NamespaceID

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: datanode, federation
    • Labels:
      None

      Description

      I'm observing inconsistencies in the DataStorage.namespaceID value passed in to my FsDatasetSpi plugin:

      • Upon initial startup (when the DataNode is formatted), the storage DataStorage.namespaceID value is populated with the namespace ID from the NameNode (or an arbitrary namenode in a federated setup).
      • Upon subsequent startups the DataStorage.namespaceID value is NULL.

      The issue appears to be different handling of the DataStorage.namespaceID value in DataStorage.format() vs. DataStorage.setFieldsFromProperties():

      DataStorage.java
        void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid) throws IOException {
          sd.clearDirectory(); // create directory
          this.layoutVersion = HdfsConstants.LAYOUT_VERSION;
          this.clusterID = nsInfo.getClusterID();
          this.namespaceID = nsInfo.getNamespaceID();
        }
      ....
      
        protected void setFieldsFromProperties(Properties props, StorageDirectory sd)
          // Read NamespaceID in version before federation
          if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
            setNamespaceID(props, sd);
          }
      

      If I understand correctly, the proper fix is to leave namespaceID unset in DataStorage.format(). The namespaceID here is invalid anyway since it is susceptible to race conditions in federated NameNode setups – it will happen to match the namespaceID of the first BPOfferService thread that successfully handshakes.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Eric Sirianni
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development