[HBASE-18998] processor.getRowsToLock() always assumes there is some row being locked

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 1.3.2, 1.1.13, 2.0.0-alpha-4, 2.0.0, 1.2.7
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 12/Oct/17 20:23
• Updated: 14/Aug/18 01:43
• Resolved: 13/Oct/17 15:48

Description

During testing, we observed the following exception:

2017-10-12 02:52:26,683|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|1/1          DROP TABLE testTable;
2017-10-12 02:52:30,320|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|17/10/12 02:52:30 WARN ipc.CoprocessorRpcChannel: Call failed on IOException
2017-10-12 02:52:30,320|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: TESTTABLE: null
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:93)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:1671)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:14347)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7849)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1980)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1962)
2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|Caused by: java.util.NoSuchElementException
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at java.util.Collections$EmptyIterator.next(Collections.java:4189)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.processRowsWithLocks(HRegion.java:7137)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:6980)
2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.mutateRowsWithLocks(MetaDataEndpointImpl.java:1966)
2017-10-12 02:52:30,323|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:1650)

Here is code from branch-1.1 :

        if (!mutations.isEmpty() && !walSyncSuccessful) {
          LOG.warn("Wal sync failed. Roll back " + mutations.size() +
              " memstore keyvalues for row(s):" + StringUtils.byteToHexString(
              processor.getRowsToLock().iterator().next()) + "...");

The assumption that processor.getRowsToLock().iterator() would always be non-empty was wrong.

In other branches, taking the iterator seems to have the same issue.

Thanks to elserj who spotted this issue.