[HDFS-16044] Fix getListing call getLocatedBlocks even source is a directory

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: namenode
• Assignee: ludun Assign to me
• Reporter: ludun
• Created: 27/May/21 08:47
• Updated: 09/Jan/22 08:53

Description

In production cluster when call getListing very frequent. The processing time of rpc request is very high. we try to optimize the performance of getListing request.
After some check, we found that, even the source and child is dir, the getListing request also call getLocatedBlocks.

the request is and needLocation is false

2021-05-27 15:16:07,093 TRACE ipc.ProtobufRpcEngine: 1: Call -> 8-5-231-4/8.5.231.4:25000: getListing {src: "/data/connector/test/topics/102test" startAfter: "" needLocation: false}

but getListing request 1000 times getLocatedBlocks which not needed.

`---ts=2021-05-27 14:19:15;thread_name=IPC Server handler 86 on 25000;id=e6;is_daemon=true;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@5fcfe4b2
    `---[35.068532ms] org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:getListing()
        +---[0.003542ms] org.apache.hadoop.hdfs.server.namenode.INodesInPath:getPathComponents() #214
        +---[0.003053ms] org.apache.hadoop.hdfs.server.namenode.FSDirectory:isExactReservedName() #95
        +---[0.002938ms] org.apache.hadoop.hdfs.server.namenode.FSDirectory:readLock() #218
        +---[0.00252ms] org.apache.hadoop.hdfs.server.namenode.INodesInPath:isDotSnapshotDir() #220
        +---[0.002788ms] org.apache.hadoop.hdfs.server.namenode.INodesInPath:getPathSnapshotId() #223
        +---[0.002905ms] org.apache.hadoop.hdfs.server.namenode.INodesInPath:getLastINode() #224
        +---[0.002785ms] org.apache.hadoop.hdfs.server.namenode.INode:getStoragePolicyID() #230
        +---[0.002236ms] org.apache.hadoop.hdfs.server.namenode.INode:isDirectory() #233
        +---[0.002919ms] org.apache.hadoop.hdfs.server.namenode.INode:asDirectory() #242
        +---[0.003408ms] org.apache.hadoop.hdfs.server.namenode.INodeDirectory:getChildrenList() #243
        +---[0.005942ms] org.apache.hadoop.hdfs.server.namenode.INodeDirectory:nextChild() #244
        +---[0.002467ms] org.apache.hadoop.hdfs.util.ReadOnlyList:size() #245
        +---[0.005481ms] org.apache.hadoop.hdfs.server.namenode.FSDirectory:getLsLimit() #247
        +---[0.002176ms] org.apache.hadoop.hdfs.server.namenode.FSDirectory:getLsLimit() #248
        +---[min=0.00211ms,max=0.005157ms,total=2.247572ms,count=1000] org.apache.hadoop.hdfs.util.ReadOnlyList:get() #252
        +---[min=0.001946ms,max=0.005411ms,total=2.041715ms,count=1000] org.apache.hadoop.hdfs.server.namenode.INode:isSymlink() #253
        +---[min=0.002176ms,max=0.005426ms,total=2.264472ms,count=1000] org.apache.hadoop.hdfs.server.namenode.INode:getLocalStoragePolicyID() #254
        +---[min=0.002251ms,max=0.006849ms,total=2.351935ms,count=1000] org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:getStoragePolicyID() #95
        +---[min=0.006091ms,max=0.012333ms,total=6.439434ms,count=1000] org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp:createFileStatus() #257
        +---[min=0.00269ms,max=0.004995ms,total=2.788194ms,count=1000] org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus:getLocatedBlocks() #265
        +---[0.003234ms] org.apache.hadoop.hdfs.protocol.DirectoryListing:<init>() #274
        `---[0.002457ms] org.apache.hadoop.hdfs.server.namenode.FSDirectory:readUnlock() #277

Issue Links

• is caused by: HDFS-12681 Make HdfsLocatedFileStatus a subtype of LocatedFileStatus