Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18810

TestClientScannerRPCTimeout failing in branch-1 / branch-1.4

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0, 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      TestClientScannerRPCTimeout is failing in branch-1 / branch-1.4, but it's a setup problem involving CreateTableProcedure. Fails for me the first time in both Linux and MacOS dev environments.

      Bisect says this is the cause, been there for a while

      commit af359d03b5e2cc798cee8ba52d2a9fcbb1022104
      Author: Stephen Yuan Jiang <syuanjiangdev@gmail.com>
      Date:   Tue Jul 18 06:58:29 2017 -0700
      
          HBASE-16488 Starting namespace and quota services in master startup asynchronizely (Stephen Yuan Jiang)
      

      and sure enough, if I revert HBASE-16488 from branch-1 / branch-1.4 then the test passes again.

      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.211 sec <<< FAILURE! - in org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout
      testScannerNextRPCTimesout(org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout)  Time elapsed: 6.248 sec  <<< ERROR!
      org.apache.hadoop.hbase.TableExistsException: testScannerNextRPCTimesout
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
              at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
              at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
              at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4774)
              at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4732)
              at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4665)
              at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:679)
              at org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1500)
              at org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1547)
              at org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1438)
              at org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1414)
              at org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1370)
              at org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout.testScannerNextRPCTimesout(TestClientScannerRPCTimeout.java:87)
      Caused by: org.apache.hadoop.ipc.RemoteException: testScannerNextRPCTimesout
              at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:286)
              at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:107)
              at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:59)
              at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
              at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:506)
              at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1152)
              at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:940)
              at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:893)
              at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76)
              at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:478)
      

        Issue Links

          Activity

          Hide
          apurtell Andrew Purtell added a comment -

          Oddly a bisect traces this failure to HBASE-16488, and if I revert that commit from branch-1 / branch-1.4, then the test passes again. I find that HBASE-16488 is still in PA state even though the change was committed at least to implicated branches, so I am inclined to revert the change given that irregularity.

          Show
          apurtell Andrew Purtell added a comment - Oddly a bisect traces this failure to HBASE-16488 , and if I revert that commit from branch-1 / branch-1.4, then the test passes again. I find that HBASE-16488 is still in PA state even though the change was committed at least to implicated branches, so I am inclined to revert the change given that irregularity.
          Hide
          apurtell Andrew Purtell added a comment -

          I'm checking for impact of the revert on other tests, but I plan to revert.

          Show
          apurtell Andrew Purtell added a comment - I'm checking for impact of the revert on other tests, but I plan to revert.
          Hide
          apurtell Andrew Purtell added a comment -

          Fixed by revert of HBASE-16488

          Show
          apurtell Andrew Purtell added a comment - Fixed by revert of HBASE-16488
          Hide
          chia7712 Chia-Ping Tsai added a comment -

          The error is due to the retry mechanism. If the first quest is timeout, the client will retry with different group/nonce. And then, the TableExistsException occurs if the second request is executed after first request has be finished. This bug can be reproduced by making MasterObserver#postCreateTable() sleep.

          Show
          chia7712 Chia-Ping Tsai added a comment - The error is due to the retry mechanism. If the first quest is timeout, the client will retry with different group/nonce. And then, the TableExistsException occurs if the second request is executed after first request has be finished. This bug can be reproduced by making MasterObserver#postCreateTable() sleep.
          Hide
          chia7712 Chia-Ping Tsai added a comment -

          I have attached the patch in HBASE-18415. Andrew Purtell Please take a look if you have free time. Thanks.

          Show
          chia7712 Chia-Ping Tsai added a comment - I have attached the patch in HBASE-18415 . Andrew Purtell Please take a look if you have free time. Thanks.

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              apurtell Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development