Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2225

"Error in coprocessor" problem while executing TOP N query

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Bug
    • v1.5.4.1
    • None
    • Storage - HBase
    • None

    Description

      Software information:

      • Kylin: apache-kylin-1.5.4.1-cdh5.7
      • Hadoop: CHD 5.7
      • HBase: 1.2.0-cdh5.7.1
      • HBase coprocessor: kylin-coprocessor-1.5.4.1-0.jar

      I successfully built a cube, but have problem while executing a TOP N query.

      Cube

      Here is the SQL statement:

      ```
      select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100
      ```

      Error message:

      ```
      Error while executing SQL "select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100": Error in coprocessor
      ```

      Log details:

      ```
      2016-11-23 15:01:56,674 WARN [kylin-coproc--pool2-t16] ipc.CoprocessorRpcChannel:58 : Call failed on IOException
      java.net.SocketTimeoutException: callTimeout=60000, callDuration=69049: row '' on table 'KYLIN_O8HMHDKRD0' at region=KYLIN_O8HMHDKRD0,,1479880633595.015f60fd1695b68f59981e1ba3c7d5fd., hostname=ym-094,60020,1479183824861, seqNum=11
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
      at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4013)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:179)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:175)
      at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1797)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Call to ym-094/172.16.24.94:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:291)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
      at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1620)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 10 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:73)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
      ... 17 more
      2016-11-23 15:01:56,678 WARN [pool-9-thread-15] client.HTable:1811 : Error calling coprocessor service org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService for row \x00\x00
      java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: callTimeout=60000, callDuration=69049: row '' on table 'KYLIN_O8HMHDKRD0' at region=KYLIN_O8HMHDKRD0,,1479880633595.015f60fd1695b68f59981e1ba3c7d5fd., hostname=ym-094,60020,1479183824861, seqNum=11
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1809)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:174)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=69049: row '' on table 'KYLIN_O8HMHDKRD0' at region=KYLIN_O8HMHDKRD0,,1479880633595.015f60fd1695b68f59981e1ba3c7d5fd., hostname=ym-094,60020,1479183824861, seqNum=11
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
      at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4013)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:179)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:175)
      at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1797)
      ... 4 more
      Caused by: java.io.IOException: Call to ym-094/172.16.24.94:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:291)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
      at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1620)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 10 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:73)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
      ... 17 more
      2016-11-23 15:01:56,679 ERROR [pool-9-thread-15] v2.CubeHBaseEndpointRPC:212 : <sub-thread for GTScanRequest 6979cfa6> Error when visiting cubes by endpoint
      java.net.SocketTimeoutException: callTimeout=60000, callDuration=69049: row '' on table 'KYLIN_O8HMHDKRD0' at region=KYLIN_O8HMHDKRD0,,1479880633595.015f60fd1695b68f59981e1ba3c7d5fd., hostname=ym-094,60020,1479183824861, seqNum=11
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
      at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4013)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:179)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:175)
      at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1797)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Call to ym-094/172.16.24.94:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:291)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
      at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1620)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 10 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:73)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
      ... 17 more
      2016-11-23 15:02:06,680 ERROR [http-bio-7070-exec-8] controller.QueryController:208 : Exception when execute sql
      java.sql.SQLException: Error while executing SQL "select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100": Error in coprocessor
      at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
      at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
      at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:147)
      at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:208)
      at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:408)
      at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:320)
      at org.apache.kylin.rest.service.QueryService.query(QueryService.java:127)
      at org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:191)
      at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
      at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
      at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
      at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
      at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
      at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Error in coprocessor
      at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:95)
      at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:37)
      at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
      at com.google.common.collect.Iterators$5.hasNext(Iterators.java:597)
      at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:96)
      at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596)
      at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:84)
      at org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
      at Baz$1$1.moveNext(Unknown Source)
      at org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:823)
      at org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:758)
      at org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
      at Baz.bind(Unknown Source)
      at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327)
      at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:282)
      at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
      at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
      at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:187)
      at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:65)
      at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
      at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:605)
      at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:599)
      at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:613)
      at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:139)
      ... 77 more
      Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=69049: row '' on table 'KYLIN_O8HMHDKRD0' at region=KYLIN_O8HMHDKRD0,,1479880633595.015f60fd1695b68f59981e1ba3c7d5fd., hostname=ym-094,60020,1479183824861, seqNum=11
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
      at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4013)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:179)
      at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:175)
      at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1797)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      ... 1 more
      Caused by: java.io.IOException: Call to ym-094/172.16.24.94:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:291)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
      at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1620)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
      at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      ... 10 more
      Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=406, waitTime=60001, operationTimeout=60000 expired.
      at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:73)
      at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
      ... 17 more
      2016-11-23 15:02:06,686 INFO [http-bio-7070-exec-8] service.QueryService:259 :
      ==========================[QUERY]===============================
      SQL: select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100
      User: ADMIN
      Success: false
      Duration: 0.0
      Project: tm_dress_data
      Realization Names: [basic_dress_daily_deal_cube2]
      Cuboid Ids: [1]
      Total scan count: 0
      Result row count: 0
      Accept Partial: true
      Is Partial Result: false
      Hit Exception Cache: false
      Storage cache used: false
      Message: Error while executing SQL "select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100": Error in coprocessor
      ==========================[QUERY]===============================

      2016-11-23 15:02:06,690 ERROR [http-bio-7070-exec-8] controller.BasicController:44 :
      org.apache.kylin.rest.exception.InternalErrorException: Error while executing SQL "select spu_id, sum(sales) from daily_deal_wide_view group by spu_id order by sum(sales) desc limit 100": Error in coprocessor
      at org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:225)
      at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
      at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
      at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
      at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
      at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
      at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:745)
      ```

      Attachments

        Activity

          People

            shaofengshi Shao Feng Shi
            liyangliang Yangliang Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: