Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13821

RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 2.9.1, 3.0.3
    • 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
    • hdfs
    • None
    • Reviewed

    Description

      When i test rbf, if found performance problem.
      I found that ProxyAvgTime From Ganglia is so high, i run jstack on Router and get the following stack frames

         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000005c264acd8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
              at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2249)
              at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
              at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
              at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
              at org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:380)
              at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2104)
              at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2087)
              at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getListing(RouterRpcServer.java:1050)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:640)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)

      Many threads blocked on LocalCache
      After disable the cache, ProxyAvgTime is down as follow showed

      Attachments

        1. LocalCacheTest.java
          2 kB
          Hui Fei
        2. image-2018-08-13-11-27-49-023.png
          228 kB
          Hui Fei
        3. HDFS-13821.008.patch
          8 kB
          Hui Fei
        4. HDFS-13821.007.patch
          8 kB
          Hui Fei
        5. HDFS-13821.006.patch
          8 kB
          Hui Fei
        6. HDFS-13821.005.patch
          8 kB
          Hui Fei
        7. HDFS-13821.004.patch
          8 kB
          Hui Fei
        8. HDFS-13821.003.patch
          6 kB
          Hui Fei
        9. HDFS-13821.002.patch
          7 kB
          Hui Fei
        10. HDFS-13821.001.patch
          6 kB
          Hui Fei

        Issue Links

          Activity

            People

              ferhui Hui Fei
              ferhui Hui Fei
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: