Details
Major Description
Phoenix or HBase keeps throwing a timeout exception. I have tryed every configuration I could think about to increase it.
Partial stacktrace:
Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
... 13 more
The Phoenix (hbase-site.xml) properties:
- phoenix.query.timeoutMs
- phoenix.query.keepAliveMs
I've tryed editing HBase config files and also setting config in Ambari with the next keys to increase the timeout with no success:
- hbase.rpc.timeout
- dfs.socket.timeout
- dfs.client.socket-timeout
- zookeeper.session.timeout
Full stack trace:
Error: Encountered exception in sub plan [0] execution. (state=,code=0)
java.sql.SQLException: Encountered exception in sub plan [0] execution.
at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:157)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:542)
at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:106)
at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:82)
at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:339)
at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:136)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
... 11 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:56)
at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:104)
at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:97)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:85)
... 5 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:223)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:50)
... 11 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5., hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
... 3 more
Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
... 13 more