[HDFS-15486] Costly sendResponse operation slows down async editlog handling

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.0
• Assignee: Unassigned
• Reporter: Yiqun Lin
• Created: 21/Jul/20 09:35
• Updated: 01/Oct/21 04:22

Description

When our cluster NameNode in a very high load, we find it often stuck in Async-editlog handling.

We use async-profile tool to get the flamegraph.

This happened in that async editlog thread consumes Edit from the queue and triggers the sendResponse call.

But here the sendResponse call is a little expensive since our cluster enabled the security env and will do some encode operations when doing the return response operation.

We often catch some moments of costly sendResponse operation when rpc call queue is fulled.

Slowness on consuming Edit in async editlog will make Edit pending Queue easily become the fulled state, then block its enqueue operation that is invoked in writeLock type methods in FSNamesystem class.

Here the enhancement is that we can use multiple thread to parallel execute sendResponse call. sendResponse doesn't need use the write lock to do protection, so this change is safe.

Issue Links

• is caused by: HDFS-15869 Network issue while FSEditLogAsync is executing RpcEdit.logSyncNotify can cause the namenode to hang
• relates to: HDFS-7964 Add support for async edit logging
• relates to: HDFS-16134 Optimize RPC throughput