HBase
  1. HBase
  2. HBASE-9087

Handlers being blocked during reads

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.7, 0.95.1
    • Fix Version/s: 0.98.0, 0.95.2, 0.94.11
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration.

      Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap.

      Here is a stack trace:

      "IPC Server handler 99 on 60020" daemon prio=10 tid=0x0000000041c84000 nid=0x2244 waiting on condition [0x00007ff51fefd000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00000000c5c13ae8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553)
        at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221)
        at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:138)
        at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3755)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771)
        at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776)
        at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
      1. HBASE-9087-1.patch
        2 kB
        Elliott Clark
      2. HBASE-9087-0.patch
        2 kB
        Elliott Clark

        Activity

        Pablo Medina created issue -
        Elliott Clark made changes -
        Field Original Value New Value
        Attachment HBASE-9087-0.patch [ 12595006 ]
        Elliott Clark made changes -
        Assignee Elliott Clark [ eclark ]
        Elliott Clark made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.95.1 [ 12324288 ]
        Elliott Clark made changes -
        Attachment HBASE-9087-1.patch [ 12595049 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.98.0 [ 12323143 ]
        Fix Version/s 0.95.2 [ 12320040 ]
        Fix Version/s 0.94.11 [ 12324741 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.12 [ 12324790 ]
        Fix Version/s 0.94.11 [ 12324741 ]
        Priority Critical [ 2 ] Major [ 3 ]
        Elliott Clark made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.11 [ 12324741 ]
        Fix Version/s 0.94.12 [ 12324790 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Elliott Clark
            Reporter:
            Pablo Medina
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development