Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15900

RS stuck in get lock of HStore

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.1.1, 1.3.0
    • None
    • None
    • None

    Description

      It happens on my production cluster when i run MR job. I save the dump.txt from this RS webUI.

      Many threads stuck here:

      Thread 133 (B.defaultRpcServer.handler=94,queue=4,port=16020):
         32   State: WAITING
         31   Blocked count: 477816
         30   Waited count: 535255
         29   Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6447ba67
         28   Stack:
         27     sun.misc.Unsafe.park(Native Method)
         26     java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
         25     java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         24     java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         23     java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         22     java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         21     org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:666)
         20     org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3621)
         19     org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3038)
         18     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2793)
         17     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2735)
         16     org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
         15     org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
         14     org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2029)
         13     org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
         12     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
         11     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
         10     org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
          9     org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
          8     java.lang.Thread.run(Thread.java:745)
      

      Attachments

        1. 0d32a6bab354e6cc170cd59a2d485797.jstack.txt
          225 kB
          Heng Chen
        2. 0d32a6bab354e6cc170cd59a2d485797.rs.log
          50 kB
          Heng Chen
        3. 9fe15a52_9fe15a52_save
          318 kB
          Heng Chen
        4. c91324eb_81194e359707acadee2906ffe36ab130.log
          375 kB
          Heng Chen
        5. dump.txt
          630 kB
          Heng Chen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            chenheng Heng Chen
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment