[HDFS-13138] webhdfs of federated namenode does not work properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 2.7.1, 3.0.0
• Component/s: webhdfs
• Assignee: Unassigned
• Reporter: KWON BYUNGCHANG
• Created: 13/Feb/18 07:27
• Updated: 31/Jan/19 04:30
• Resolved: 31/Jan/19 04:30

Description

my cluster has multiple namenodes using HDFS Federation.

webhdfs that is not defaultFS does not work properly.

when I uploaded to non defaultFS namenode  using webhdfs.

uploaded file was founded at defaultFS namenode.

 

I think root cause is that

  clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.

 https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462

 

/**
   * Set the namenode address that will be used by clients to access this
   * namenode or name service. This needs to be called before the config
   * is overriden.
   */
  public void setClientNamenodeAddress(Configuration conf) {
    String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
    if (nnAddr == null) {
      // default fs is not set.
      clientNamenodeAddress = null;
      return;
    }

    LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
    URI nnUri = URI.create(nnAddr);

    String nnHost = nnUri.getHost();
    if (nnHost == null) {
      clientNamenodeAddress = null;
      return;
    }

    if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
      // host name is logical
      clientNamenodeAddress = nnHost;
    } else if (nnUri.getPort() > 0) {
      // physical address with a valid port
      clientNamenodeAddress = nnUri.getAuthority();
    } else {
      // the port is missing or 0. Figure out real bind address later.
      clientNamenodeAddress = null;
      return;
    }
    LOG.info("Clients are to use {} to access"
        + " this namenode/service.", clientNamenodeAddress );
  }

 

so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter

finally file was located namenode of fs,defaultFS

 

workaround is

  configure fs.defaultFS of each namenode to its own nameservice.  

e.g.

  hdfs://ns1  has fs.defaultFS=hdfs://ns1

  hdfs://ns2  has fs.defaultFS=hdfs://ns2

  ....

 

 

 

 

Issue Links

• is duplicated by: HDFS-13433 webhdfs requests can be routed incorrectly in federated cluster