Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4685

Properly handle connection caching for Phoenix inside RegionServers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.14.0, 5.0.0
    • Labels:
      None

      Description

      Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. This is happening even with short circuit writes enabled.

      2018-04-08 13:06:04,747 WARN  [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed
      java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
              at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
              at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
              at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
              at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
              at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
              at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
              at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
              at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
              at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
              at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
              at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
              at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
              at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
              at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
              at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
              at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
              at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      Caused by: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
              at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
              at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
              at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40)
              at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183)
               ... 25 more
      Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread
              at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266)
              at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
              ... 32 more
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
              at java.lang.Thread.start0(Native Method)
              at java.lang.Thread.start(Thread.java:714)
              at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
              at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
              at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
              at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007)
              at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986)
              at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724)
              at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709)
              at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236)
              ... 36 more
      

        Attachments

        1. PHOENIX-4685_5.x-HBase-2.0.patch
          27 kB
          Rajeshbabu Chintaguntla
        2. PHOENIX-4685_addendum.patch
          1 kB
          Rajeshbabu Chintaguntla
        3. PHOENIX-4685_addendum2.patch
          6 kB
          James R. Taylor
        4. PHOENIX-4685_addendum3.patch
          4 kB
          James R. Taylor
        5. PHOENIX-4685_addendum4.patch
          5 kB
          James R. Taylor
        6. PHOENIX-4685_jstack
          1.86 MB
          Rajeshbabu Chintaguntla
        7. PHOENIX-4685_v2.patch
          0.8 kB
          Rajeshbabu Chintaguntla
        8. PHOENIX-4685_v3.patch
          13 kB
          Rajeshbabu Chintaguntla
        9. PHOENIX-4685_v4.patch
          24 kB
          Rajeshbabu Chintaguntla
        10. PHOENIX-4685_v5.patch
          24 kB
          Rajeshbabu Chintaguntla
        11. PHOENIX-4685.patch
          0.9 kB
          Rajeshbabu Chintaguntla

          Activity

            People

            • Assignee:
              rajeshbabu Rajeshbabu Chintaguntla
              Reporter:
              rajeshbabu Rajeshbabu Chintaguntla
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: