[HDFS-12285] Better handling of namenode ip address change

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ming Ma
• Created: 09/Aug/17 22:21
• Updated: 15/Mar/18 00:12

Description

RPC client layer provides functionality to detect ip address change:

Client.java
    private synchronized boolean updateAddress() throws IOException {
      // Do a fresh lookup with the old host name.
      InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
                               server.getHostName(), server.getPort());
    ......
    }

To use this feature, we need to enable retry via dfs.client.retry.policy.enabled. Otherwise TryOnceThenFail RetryPolicy will be used; which caused handleConnectionFailure to throw ConnectException exception without retrying with the new ip address.

    private void handleConnectionFailure(int curRetries, IOException ioe
        ) throws IOException {
      closeConnection();

      final RetryAction action;
      try {
        action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
      } catch(Exception e) {
        throw e instanceof IOException? (IOException)e: new IOException(e);
      }
  ......
  }

However, using such configuration isn't ideal. What happens is DFSClient still holds onto the cached old ip address created by namenode = proxyInfo.getProxy();. Thus when a new rpc connection is created, it starts with the old ip followed by retry with the new ip. It will be nice if DFSClient can update namenode proxy automatically upon ip address change.

Issue Links

• relates to: HADOOP-12125 Retrying UnknownHostException on a proxy does not actually retry hostname resolution