[HDFS-13821] RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 2.9.1, 3.0.3
• Fix Version/s: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
• Component/s: hdfs
• Target Version/s: 2.10.0, 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Hui Fei
• Reporter: Hui Fei
• Created: 13/Aug/18 03:28
• Updated: 06/Oct/21 12:03
• Resolved: 22/Aug/18 05:20

Description

When i test rbf, if found performance problem.
I found that ProxyAvgTime From Ganglia is so high, i run jstack on Router and get the following stack frames

   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005c264acd8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2249)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
        at org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:380)
        at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2104)
        at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2087)
        at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getListing(RouterRpcServer.java:1050)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:640)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)

Many threads blocked on LocalCache
After disable the cache, ProxyAvgTime is down as follow showed

Issue Links

• is related to: HDFS-16257 [HDFS] [RBF] Guava cache performance issue in Router MountTableResolver