[HDFS-16134] Optimize RPC throughput

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: JiangHua Zhu
• Reporter: JiangHua Zhu
• Created: 21/Jul/21 07:44
• Updated: 16/Aug/21 09:16
• Resolved: 16/Aug/21 09:16

Description

Now RPC has the following phenomena:
1. Each RPC has 1 Responder, which is used to return the result, but many client requests are directly returned after the handler is processed;
2. Call#Connection#responseQueue uses synchronous control every time it is used;
These phenomena will hinder the performance of RPC, we can try to make some changes.
1. Set multiple responders for each RPC, and the user returns the result;
2. After the Handler processing is completed, add the results that need to be returned to a queue, and the Responder will process the queue data;
3. Add an identifier similar to sequenceId to ensure the order of return;

Issue Links

• is related to: HDFS-15486 Costly sendResponse operation slows down async editlog handling
• is related to: HDFS-15869 Network issue while FSEditLogAsync is executing RpcEdit.logSyncNotify can cause the namenode to hang