[HDFS-16388] The namenode should check the connection channel before the RPC returns data

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.2
• Component/s: namanode
• Assignee: Unassigned
• Reporter: JiangHua Zhu
• Created: 17/Dec/21 03:20
• Updated: 17/Dec/21 03:23

Description

Some RBF-related information has been reported in HDFS-15078 and HDFS-15079, which may be similar to the situation here.
When the client connects to the NameNode, such as creating a file, sometimes RPC will prompt some exception information:
2021-12-15 14:57:14,502 [2246654536] - WARN [IPC Server handler 58 on 8025:Server$Responder@1518] - IPC Server handler 58 on 8025, call Call#870382577 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.196.73.49:33700: output error
2021-12-15 14:57:14,502 [2246654536] - INFO [IPC Server handler 58 on 8025:Server$Handler@2619] - IPC Server handler 58 on 8025 caught an exception
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3173)
at org.apache.hadoop.ipc.Server.access$1700(Server.java:136)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1468)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1538)
at org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2517)
at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1610)
at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:935)
at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:769)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:880)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606)

This information exists in the log recorded by the NameNode, which is somewhat different from HDFS-15078&HDFS-15079, but they have a common feature, that is, they all appear when accessing RPC.
I think this problem can be solved in a unified way.

Issue Links

• is related to: HDFS-15079 RBF: Client maybe get an unexpected result with network anomaly
• is related to: HDFS-15078 RBF: Should check connection channel before sending RPC to namenode