Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3202

Error trying to remove Hash Cache - Followed by CallQueueTooBigException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Problem
    • Affects Version/s: 4.7.0
    • Fix Version/s: 4.7.0
    • Component/s: None
    • Labels:
    • Environment:

      Amazon EMR - 4.7.2

    • Flags:
      Important

      Description

      This issue is in continuation to -
      https://issues.apache.org/jira/browse/PHOENIX-3200

      14:26:14.653 [pool-2-thread-46] ERROR org.apache.phoenix.cache.ServerCacheClient - Error trying to remove hash cache for region=EPOEVENT,,1471683321681.6d9e058365a24e63e5802ca783c09660., hostname=ip-172-31-8-132.us-west-2.compute.internal,60020,1471683102656, seqNum=1935501
      org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
      Tue Aug 23 14:16:03 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:03 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , java.io.IOException: Call to ip-172-31-8-132.us-west-2.compute.internal/172.31.8.132:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=949539152, waitTime=60001, operationTimeout=60000 expired.
      Tue Aug 23 14:17:03 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:04 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:06 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:10 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:21 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:31 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:41 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:17:51 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:18:11 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:18:31 UTC 2016, RpcRetryingCaller{globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?
      Tue Aug 23 14:18:51 UTC 2016, RpcRetryingCaller

      {globalStartTime=1471961763091, pause=100, retries=35}

      , org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /172.31.8.132:60020, too many items queued ?

      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService$Stub.removeServerCache(ServerCachingProtos.java:3378) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.phoenix.cache.ServerCacheClient$2.call(ServerCacheClient.java:343) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.phoenix.cache.ServerCacheClient$2.call(ServerCacheClient.java:322) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1751) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_101]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_101]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_101]
      at java.lang.Thread.run(Thread.java:745) [?:1.7.0_101]
      Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: Call queue is full on /172.31.8.132:60020, too many items queued ?
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1235) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:32675) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1624) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ~[PhoenixMultiCopyDataLoader-0.0.1-SNAPSHOT-jar-with-dependencies.jar:?]
      ... 10 more

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nvommi Nithin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: