[HDFS-13671] Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.0.3
• Fix Version/s: 3.4.0, 3.2.3, 3.3.2
• Component/s: namenode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Haibin Huang
• Reporter: Yiqun Lin
• Created: 12/Jun/18 10:19
• Updated: 16/Oct/24 18:29
• Resolved: 18/Jun/21 13:14

Description

NameNode hung when deleting large files/blocks. The stack info:

"IPC Server handler 4 on 8020" #87 daemon prio=5 os_prio=0 tid=0x00007fb505b27800 nid=0x94c3 runnable [0x00007fa861361000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.hadoop.hdfs.util.FoldedTreeSet.compare(FoldedTreeSet.java:474)
	at org.apache.hadoop.hdfs.util.FoldedTreeSet.removeAndGet(FoldedTreeSet.java:849)
	at org.apache.hadoop.hdfs.util.FoldedTreeSet.remove(FoldedTreeSet.java:911)
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:252)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:194)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:108)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlockFromMap(BlockManager.java:3813)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3617)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocks(FSNamesystem.java:4270)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4244)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4180)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4164)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:871)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:311)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:625)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

In the current deletion logic in NameNode, there are mainly two steps:

• Collect INodes and all blocks to be deleted, then delete INodes.
• Remove blocks chunk by chunk in a loop.
  Actually the first step should be a more expensive operation and will takes more time. However, now we always see NN hangs during the remove block operation.

Looking into this, we introduced a new structure FoldedTreeSet to have a better performance in dealing FBR/IBRs. But compared with early implementation in remove-block logic, FoldedTreeSet seems more slower since It will take additional time to balance tree node. When there are large block to be removed/deleted, it looks bad.

For the get type operations in DatanodeStorageInfo, we only provide the getBlockIterator to return blocks iterator and no other get operation with specified block. Still we need to use FoldedTreeSet in DatanodeStorageInfo? As we know FoldedTreeSet is benefit for Get not Update. Maybe we can revert this to the early implementation.

Issue Links

• is related to: HDFS-16059 dfsadmin -listOpenFiles -blockingDecommission can miss some files
• is related to: HDFS-15574 Remove unnecessary sort of block list in DirectoryScanner
• relates to: HDFS-15131 FoldedTreeSet appears to degrade over time
• relates to: HDFS-9260 Improve the performance and GC friendliness of NameNode startup and full block reports
• links to: GitHub Pull Request #3065
• links to: GitHub Pull Request #3113
• links to: GitHub Pull Request #3114