[HADOOP-18346] Suggest to update the IP address of ConnectionId when server of Connection got updated

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Component/s: hdfs-client
• Assignee: Unassigned
• Reporter: Shawn
• Created: 18/Jul/22 19:04
• Updated: 08/Aug/22 19:19

Description

When use ProtobufRpcEngine to get RPC proxy, we apply the InetSocketAddress addr, and this address is then kept in the ConnectionId (see here and here).

When the destination IP of the connection changes, we have code to update the server variable in the connection (see here). This makes the retry of the connection to succeed.

However, the above logic only temporary fix the connection. After the RPC call of the client, the connection will be closed. Next time when we use the same proxy to do the RPC call, we will again to fail the first connection and succeed the second one after the server IP got updates.

My suggestion is to not only update the server variable in the connection, but also update the address of the ConnectionId in the Invoker. This way, we can reuse the proxy with the latest IP address.

The use case for this is: in Apache ozone project, we initialize the RPC proxy with a fixed InetSocketAddress (for example, datanode needs to set up the RPC proxy to scm). But if lately the server IP changes (e.g. in k8s, the IP of a pod is very likely to change), every time the proxy needs two calls (1st fails and the 2nd succeeds with the updated IP). By using the above solution, we can eliminate this issue.

Issue Links

• blocks: HDDS-7004 When SCM IP changes, all the other ozone components keep generating WARN logs
• duplicates: HADOOP-18365 Updated addresses are still accessed using the old IP address