Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4685

RpcException: Data not accepted downstream

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • 1.7.0
    • None
    • Execution - RPC
    • None

    Description

      commit # : d93a3633815ed1c7efd6660eae62b7351a2c9739

      Scenario : Each of the below 2 queries are duplicated 10 times to make a total of 20 queries. Now I use 10 concurrent clients to run the queries.

      Query 1 :

      select count(*) from (
        select max(length(concat(str1,str2))) max_length from (
          select
              substring(regexp_replace(s.enlarged_comment, 'iron..', 'iconic'), 4) str1,
              substring(regexp_replace(s.enlarged_comment, 'm.*ne', 'iconic'), 4) str2
          from (
              select
                  concat(o_comment, o_comment, o_comment, o_comment, o_comment, o_comment,      o_comment, o_comment, o_comment, o_comment, o_comment, o_comment, o_comment,  o_comment, o_comment) enlarged_comment,
                  o_orderdate,
                  concat(o_clerk, '...') o_clerk
              from orders_nocompression_256
              where o_orderdate > date '1900-01-01' + interval '1' year
            ) s
            where
                position('...' in o_clerk) > 0
                and length(concat(enlarged_comment, o_clerk)) > 100
            limit 5000000
        ) s1
      )s2 where max_length = 0
      

      Query 2 :

      Select  count(*)  from lineitem_nocompression_256
      where
          (
              l_tax in (0.02,0.06,0.04,0.05,0.0,0.07,0.08,0.03,0.01)
              and l_linestatus='F'
          )
          or (
              ( length(l_comment) between 0 and 50)
              and (
                  substr(l_shipmode, 1,2)='R'
                  or substr(l_shipmode, 1,2)='A'
                  and l_tax > 0.05
              )
          )
          or (
              (  l_extendedprice between 10000.0 and 100000.0 )
              and l_linestatus='O'
          )
          or (
              l_extendedprice*l_discount*l_tax < 45.00
              and l_shipdate > date '1996-03-13' + interval '1' year
          )
          or (
              l_commitdate in (
                  date '1996-02-12', date '1996-02-28', date '1996-03-05', date '1996-03-30', date '1996-03-14', date '1996-02-07', date '1997-01-14', date '1994-01-04'
              )
              and l_tax in (
                  0.02,0.06,0.04,0.05,0.0,0.07,0.08,0.03,0.01
              )
              and length(l_comment) > 15
          )
          or (
              position('con' in regexp_replace(l_comment, 'm.*ne', 'iconic')) > 10
              and (
                  length(regexp_replace(concat(lower(l_shipinstruct), lower(l_shipmode), l_comment), 'd.*ne', '')) > 0
                  or l_orderkey>50000
                  or l_partkey>1500
                  or l_linenumber=7
              )
          );
      

      Out of the 20 queries submitted 2 copies of the first query failed with the same error. Below is the summary of the run

      PASS (8.185 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 147182110)
      PASS (8.191 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 671870783)
      PASS (8.287 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 640915121)
      PASS (8.444 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 778960233)
      PASS (9.232 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 2022177583)
      PASS (2.423 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 778960233)
      PASS (11.67 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 1673732733)
      PASS (2.693 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1673732733)
      PASS (15.32 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1651684372)
      [#25] Query failed: 
      oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: RpcException: Data not accepted downstream.
      
      Fragment 2:43
      
      [Error Id: 04168b77-dfdd-4e6c-9e86-33317c82947b on atsqa6c81.qa.lab:31010]
      	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
      	at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:114)
      	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
      	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
      	at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:65)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:363)
      	at oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:240)
      	at oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:245)
      	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
      	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      	at java.lang.Thread.run(Thread.java:744)
      [#26] Query failed: 
      oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: RpcException: Data not accepted downstream.
      
      Fragment 2:11
      
      [Error Id: a155311b-3e50-44fe-94b5-2a258148ca4e on atsqa6c81.qa.lab:31010]
      	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
      	at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:114)
      	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
      	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
      	at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:65)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:363)
      	at oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:240)
      	at oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
      	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:245)
      	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
      	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      	at java.lang.Thread.run(Thread.java:744)
      EXECUTION_FAILURE (18.42 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 524239451)
      EXECUTION_FAILURE (18.43 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 297200484)
      PASS (12.53 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 671870783)
      PASS (9.851 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 778960233)
      PASS (20.73 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1166215941)
      PASS (3.144 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 524239451)
      PASS (16.74 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 147182110)
      PASS (11.74 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 1673732733)
      PASS (16.87 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 2022177583)
      PASS (17.82 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 640915121)
      PASS (10.98 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1651684372)
      

      Attachments

        1. instance2-profile.json
          125 kB
          Rahul Kumar Challapalli
        2. instance1-profile.json
          125 kB
          Rahul Kumar Challapalli
        3. drillbit-sender.log
          367 kB
          Rahul Kumar Challapalli
        4. drillbit-receiver.log
          341 kB
          Rahul Kumar Challapalli

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rkins Rahul Kumar Challapalli
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: