Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18998

processor.getRowsToLock() always assumes there is some row being locked

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      During testing, we observed the following exception:

      2017-10-12 02:52:26,683|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|1/1          DROP TABLE testTable;
      2017-10-12 02:52:30,320|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|17/10/12 02:52:30 WARN ipc.CoprocessorRpcChannel: Call failed on IOException
      2017-10-12 02:52:30,320|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: TESTTABLE: null
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:93)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:1671)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:14347)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7849)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1980)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1962)
      2017-10-12 02:52:30,321|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|Caused by: java.util.NoSuchElementException
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at java.util.Collections$EmptyIterator.next(Collections.java:4189)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.processRowsWithLocks(HRegion.java:7137)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:6980)
      2017-10-12 02:52:30,322|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.mutateRowsWithLocks(MetaDataEndpointImpl.java:1966)
      2017-10-12 02:52:30,323|INFO|MainThread|machine.py:164 - run()||GUID=f4cd2a25-3040-41cc-b423-9ec7990048f4|at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:1650)
      

      Here is code from branch-1.1 :

              if (!mutations.isEmpty() && !walSyncSuccessful) {
                LOG.warn("Wal sync failed. Roll back " + mutations.size() +
                    " memstore keyvalues for row(s):" + StringUtils.byteToHexString(
                    processor.getRowsToLock().iterator().next()) + "...");
      

      The assumption that processor.getRowsToLock().iterator() would always be non-empty was wrong.

      In other branches, taking the iterator seems to have the same issue.

      Thanks to Josh Elser who spotted this issue.

        Attachments

        1. 18998.v1.txt
          2 kB
          Ted Yu

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              yuzhihong@gmail.com Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: