Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-2161

Can't change timeout

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 4.4.0
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      Hadoop with Ambari 2.1.0
      Phoenix 4.4.0.2.3
      HBase 1.1.1.2.3
      HDFS 2.7.1.2.3
      Zookeeper 3.4.6.2.3

      Description

      Phoenix or HBase keeps throwing a timeout exception. I have tryed every configuration I could think about to increase it.

      Partial stacktrace:

      Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 4 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
      ... 13 more

      The Phoenix (hbase-site.xml) properties:

      • phoenix.query.timeoutMs
      • phoenix.query.keepAliveMs

      I've tryed editing HBase config files and also setting config in Ambari with the next keys to increase the timeout with no success:

      • hbase.rpc.timeout
      • dfs.socket.timeout
      • dfs.client.socket-timeout
      • zookeeper.session.timeout

      Full stack trace:

      Error: Encountered exception in sub plan [0] execution. (state=,code=0)
      java.sql.SQLException: Encountered exception in sub plan [0] execution.
      at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:157)
      at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
      at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
      at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
      at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
      at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
      at sqlline.Commands.execute(Commands.java:822)
      at sqlline.Commands.sql(Commands.java:732)
      at sqlline.SqlLine.dispatch(SqlLine.java:808)
      at sqlline.SqlLine.begin(SqlLine.java:681)
      at sqlline.SqlLine.start(SqlLine.java:398)
      at sqlline.SqlLine.main(SqlLine.java:292)
      Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
      at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
      at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:542)
      at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
      at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
      at org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:106)
      at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:82)
      at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:339)
      at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:136)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:202)
      at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
      ... 11 more
      Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
      Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
      at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
      at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:56)
      at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:104)
      at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
      at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
      at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
      at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:97)
      at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:85)
      ... 5 more
      Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
      Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
      at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:223)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
      at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
      at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
      at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:50)
      ... 11 more
      Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      ... 3 more
      Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 4 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
      ... 13 more

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              katezer AdriĆ  V.
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: