[PHOENIX-4122] Prevent IllegalMonitorStateException from occurring when releasing row locks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.12.0
• Labels: secondary_index
• Assignee: James R. Taylor
• Reporter: James R. Taylor
• Created: 24/Aug/17 21:14
• Updated: 11/Oct/17 22:28
• Resolved: 25/Aug/17 21:15

Description

In this iteration, after RS process was killed, Indexes were disabled and soon its status was changed to INACTIVE state.
After all regions were re-assigned, we observed that two RS were aborted with below exception in logs

org.apache.phoenix.hbase.index.Indexer threw java.lang.IllegalMonitorStateException
java.lang.IllegalMonitorStateException
        at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
        at org.apache.phoenix.hbase.index.LockManager$RowLockImpl.release(LockManager.java:223)
        at org.apache.phoenix.hbase.index.LockManager$RowLockContext.releaseRowLock(LockManager.java:166)
        at org.apache.phoenix.hbase.index.LockManager.unlockRow(LockManager.java:131)
        at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1023)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1621)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1697)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1653)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1019)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2837)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2410)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2365)
        at org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatch(UngroupedAggregateRegionObserver.java:225)
        at org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commit(UngroupedAggregateRegionObserver.java:791)
        at org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:705)
        at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:255)
        at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:306)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3361)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32492)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2210)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)
2017-08-24 07:47:19,916 FATAL [4,queue=4,port=60020] regionserver.HRegionServer - ABORTING region server