[HBASE-20236] [locking] Write-time worst offenders

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0-beta-2
• Component/s: Performance
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 21/Mar/18 00:21
• Updated: 22/Jan/20 03:32
• Resolved: 22/Jan/20 03:32

Description

Messing w/ my new toy, here are worst offenders locking; they must be bad if they show up in this sampling profiler:

 7 Total: 769321884622 (99.24%)  samples: 2965
  8   [ 0] java.util.concurrent.Semaphore$NonfairSync
  9   [ 1] sun.misc.Unsafe.park
 10   [ 2] java.util.concurrent.locks.LockSupport.park
 11   [ 3] java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
 12   [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
 13   [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
 14   [ 6] java.util.concurrent.Semaphore.acquire
 15   [ 7] org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
 16   [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
 17
 18 Total: 4284274263 (0.55%)  samples: 23543
 19   [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
 20   [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
 21   [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
 22   [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
 23   [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
 24   [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
 25   [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
 26   [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
 27   [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
 28   [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
 29   [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
 30   [11] org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
 31   [12] org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
 32   [13] org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
 33   [14] org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
 34   [15] org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
 35   [16] org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
 36   [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
 37   [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
 38   [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
 39   [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
 40   [21] org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
 41   [22] org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
 42   [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
 43   [24] org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
 44   [25] org.apache.hadoop.hbase.ipc.RpcServer.call
 45   [26] org.apache.hadoop.hbase.ipc.CallRunner.run
 46   [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
 47   [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
 48
 49 Total: 717708856 (0.09%)  samples: 214
 50   [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
 51   [ 1] sun.misc.Unsafe.park
 52   [ 2] java.util.concurrent.locks.LockSupport.park
 53   [ 3] java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
 54   [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
 55   [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
 56   [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
 57   [ 7] org.apache.hadoop.hbase.regionserver.HRegion.blockUpdates
 58   [ 8] org.apache.hadoop.hbase.regionserver.RegionServicesForStores.blockUpdates
 59   [ 9] org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory
 60   [10] org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run
 61   [11] java.util.concurrent.ThreadPoolExecutor.runWorker
 62   [12] java.util.concurrent.ThreadPoolExecutor$Worker.run
 63   [13] java.lang.Thread.run
...

Issue Links

• links to: Review Board (master)