[HDFS-14442] Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 3.2.2
• Assignee: Ravuri Sushma sree
• Reporter: Erik Krogen
• Created: 18/Apr/19 22:03
• Updated: 05/Oct/20 10:12
• Resolved: 12/Mar/20 13:47

Description

While working on HDFS-14245, we noticed a discrepancy in some proxy-handling code.

The description of RpcInvocationHandler.getConnectionId() states:

  /**
   * Returns the connection id associated with the InvocationHandler instance.
   * @return ConnectionId
   */
  ConnectionId getConnectionId();

It does not make any claims about whether this connection ID will be an active proxy or not. Yet in HAUtil we have:

  /**
   * Get the internet address of the currently-active NN. This should rarely be
   * used, since callers of this method who connect directly to the NN using the
   * resulting InetSocketAddress will not be able to connect to the active NN if
   * a failover were to occur after this method has been called.
   * 
   * @param fs the file system to get the active address of.
   * @return the internet address of the currently-active NN.
   * @throws IOException if an error occurs while resolving the active NN.
   */
  public static InetSocketAddress getAddressOfActive(FileSystem fs)
      throws IOException {
    if (!(fs instanceof DistributedFileSystem)) {
      throw new IllegalArgumentException("FileSystem " + fs + " is not a DFS.");
    }
    // force client address resolution.
    fs.exists(new Path("/"));
    DistributedFileSystem dfs = (DistributedFileSystem) fs;
    DFSClient dfsClient = dfs.getClient();
    return RPC.getServerAddress(dfsClient.getNamenode());
  }

Where the call RPC.getServerAddress() eventually terminates into RpcInvocationHandler#getConnectionId(), via RPC.getServerAddress() -> RPC.getConnectionIdForProxy() -> RpcInvocationHandler#getConnectionId(). HAUtil appears to be making an incorrect assumption that RpcInvocationHandler will necessarily return an active connection ID. ObserverReadProxyProvider demonstrates a counter-example to this, since the current connection ID may be pointing at, for example, an Observer NameNode.

Issue Links

• breaks: HDFS-15613 RBF: Router FSCK fails after HDFS-14442