Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9032

Impala returns 0 rows over hs2-http without waiting for fetch_rows_timeout_ms timeout

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 3.4.0
    • Fix Version/s: None
    • Component/s: Backend
    • Labels:
      None

      Description

      This looks like a bug to me but I'm not entirely sure. I'm trying to run our tests over hs2-http (IMPALA-8863) and after the change for IMPALA-7312 to introduce a non-blocking mode for FetchResults() it looks like we sometimes return an empty result way before fetch_rows_timeout_ms has elapsed. This triggers a bug in Impyla (#369), but it also seems like something we should investigate and fix in Impala.

      I1007 22:10:10.697760 56550 impala-hs2-server.cc:821] FetchResults(): query_id=764d4313dbc64e20:2831560c00000000 fetch_size=1024I1007 22:10:10.697760 56550 impala-hs2-server.cc:821] FetchResults(): query_id=764d4313dbc64e20:2831560c00000000 fetch_size=1024I1007 22:10:10.697988 56527 scheduler.cc:468] 6d4cba4d2e8ccc42:66ce26a800000000] Exec at coord is falseI1007 22:10:10.698014 54090 impala-hs2-server.cc:663] GetOperationStatus(): query_id=0d43fd73ce4403fd:da25dde900000000I1007 22:10:10.698173   127 control-service.cc:142] 0646e91fd6a0a953:02949ff300000000] ExecQueryFInstances(): query_id=0646e91fd6a0a953:02949ff300000000 coord=b04a12d76e27:22000 #instances=1I1007 22:10:10.698356 56527 admission-controller.cc:1270] 6d4cba4d2e8ccc42:66ce26a800000000] Trying to admit id=6d4cba4d2e8ccc42:66ce26a800000000 in pool_name=root.default executor_group_name=default per_host_mem_estimate=52.02 MB dedicated_coord_mem_estimate=110.02 MB max_requests=-1 (configured statically) max_queued=200 (configured statically) max_mem=29.30 GB (configured statically)I1007 22:10:10.698386 56527 admission-controller.cc:1282] 6d4cba4d2e8ccc42:66ce26a800000000] Stats: agg_num_running=9, agg_num_queued=0, agg_mem_reserved=8.34 GB,  local_host(local_mem_admitted=9.09 GB, num_admitted_running=9, num_queued=0, backend_mem_reserved=6.70 GB)I1007 22:10:10.698415 56527 admission-controller.cc:871] 6d4cba4d2e8ccc42:66ce26a800000000] Admitting query id=6d4cba4d2e8ccc42:66ce26a800000000I1007 22:10:10.698479 56527 impala-server.cc:1713] 6d4cba4d2e8ccc42:66ce26a800000000] Registering query locationsI1007 22:10:10.698529 56527 coordinator.cc:97] 6d4cba4d2e8ccc42:66ce26a800000000] Exec() query_id=6d4cba4d2e8ccc42:66ce26a800000000 stmt=select count(*) from alltypes where month=1I1007 22:10:10.698992 56527 coordinator.cc:361] 6d4cba4d2e8ccc42:66ce26a800000000] starting execution on 3 backends for query_id=6d4cba4d2e8ccc42:66ce26a800000000I1007 22:10:10.699383 56523 coordinator.cc:375] 0646e91fd6a0a953:02949ff300000000] started execution on 1 backends for query_id=0646e91fd6a0a953:02949ff300000000I1007 22:10:10.699409 56534 scheduler.cc:468] e1495f928c2cd4f6:eeda82aa00000000] Exec at coord is falseI1007 22:10:10.700017   127 control-service.cc:142] 6d4cba4d2e8ccc42:66ce26a800000000] ExecQueryFInstances(): query_id=6d4cba4d2e8ccc42:66ce26a800000000 coord=b04a12d76e27:22000 #instances=1I1007 22:10:10.700147 56534 scheduler.cc:468] e1495f928c2cd4f6:eeda82aa00000000] Exec at coord is falseI1007 22:10:10.700234   325 TAcceptQueueServer.cpp:340] New connection to server hiveserver2-http-frontend from client <Host: 172.18.0.1 Port: 51580>I1007 22:10:10.700286   329 TAcceptQueueServer.cpp:227] TAcceptQueueServer: hiveserver2-http-frontend started connection setup for client <Host: 172.18.0.1 Port: 51580>I1007 22:10:10.700314   329 TAcceptQueueServer.cpp:245] TAcceptQueueServer: hiveserver2-http-frontend finished connection setup for client <Host: 172.18.0.1 Port: 51580>I1007 22:10:10.700371 56550 impala-hs2-server.cc:844] FetchResults(): query_id=764d4313dbc64e20:2831560c00000000 #results=1 has_more=trueI1007 22:10:10.700508 56551 impala-server.cc:1969] Connection 8249c7defcb10124:1bc65ed9ea562aab from client 172.18.0.1:51576 to server hiveserver2-http-frontend closed. The connection had 1 associated session(s).I1007 22:10:10.700688 53748 impala-beeswax-server.cc:260] close(): query_id=e9473ff80c5d4afe:733cefe000000000I1007 22:10:10.700711 53748 impala-server.cc:1129] UnregisterQuery(): query_id=e9473ff80c5d4afe:733cefe000000000I1007 22:10:10.700721 53748 impala-server.cc:1234] Cancel(): query_id=e9473ff80c5d4afe:733cefe000000000I1007 22:10:10.700742 53748 coordinator.cc:716] CancelBackends() query_id=e9473ff80c5d4afe:733cefe000000000, tried to cancel 0 backendsI1007 22:10:10.700690 56534 scheduler.cc:468] e1495f928c2cd4f6:eeda82aa00000000] Exec at coord is falseI1007 22:10:10.701387 56534 admission-controller.cc:1270] e1495f928c2cd4f6:eeda82aa00000000] Trying to admit id=e1495f928c2cd4f6:eeda82aa00000000 in pool_name=root.default executor_group_name=default per_host_mem_estimate=231.95 MB dedicated_coord_mem_estimate=109.92 MB max_requests=-1 (configured statically) max_queued=200 (configured statically) max_mem=29.30 GB (configured statically)I1007 22:10:10.701417 56534 admission-controller.cc:1282] e1495f928c2cd4f6:eeda82aa00000000] Stats: agg_num_running=10, agg_num_queued=0, agg_mem_reserved=8.34 GB,  local_host(local_mem_admitted=9.84 GB, num_admitted_running=10, num_queued=0, backend_mem_reserved=6.70 GB)I1007 22:10:10.701447 56534 admission-controller.cc:871] e1495f928c2cd4f6:eeda82aa00000000] Admitting query id=e1495f928c2cd4f6:eeda82aa00000000I1007 22:10:10.701503 56534 impala-server.cc:1713] e1495f928c2cd4f6:eeda82aa00000000] Registering query locationsI1007 22:10:10.701534 56534 coordinator.cc:97] e1495f928c2cd4f6:eeda82aa00000000] Exec() query_id=e1495f928c2cd4f6:eeda82aa00000000 stmt=with t1 as (select int_col x, bigint_col y from functional.alltypes limit 2),           t2 as (select int_col x, bigint_col y from functional.alltypestiny limit 2),           t3 as (select int_col x, bigint_col y from functional.alltypessmall limit 2)      select * from t1, t2, t3 where t1.x = t2.x and t2.x = t3.xI1007 22:10:10.702641 56534 coordinator.cc:361] e1495f928c2cd4f6:eeda82aa00000000] starting execution on 3 backends for query_id=e1495f928c2cd4f6:eeda82aa00000000I1007 22:10:10.703843 56556 query-state.cc:643] 6d4cba4d2e8ccc42:66ce26a800000000] Executing instance. instance_id=6d4cba4d2e8ccc42:66ce26a800000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=2 #in-flight=18I1007 22:10:10.704418 56550 impala-server.cc:1969] Connection 41442926b263eea1:b922dd2b3a2513bd from client 172.18.0.1:51572 to server hiveserver2-http-frontend closed. The connection had 1 associated session(s).I1007 22:10:10.704545   325 TAcceptQueueServer.cpp:340] New connection to server hiveserver2-http-frontend from client <Host: 172.18.0.1 Port: 51584>I1007 22:10:10.704847 56553 query-state.cc:643] 0646e91fd6a0a953:02949ff300000000] Executing instance. instance_id=0646e91fd6a0a953:02949ff300000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=19I1007 22:10:10.705631   327 TAcceptQueueServer.cpp:227] TAcceptQueueServer: hiveserver2-http-frontend started connection setup for client <Host: 172.18.0.1 Port: 51584>I1007 22:10:10.705659   327 TAcceptQueueServer.cpp:245] TAcceptQueueServer: hiveserver2-http-frontend finished connection setup for client <Host: 172.18.0.1 Port: 51584>I1007 22:10:10.705934 56555 impala-server.cc:1969] Connection 0b41c31be5df55d6:c2d9fcea092c26b2 from client 172.18.0.1:51580 to server hiveserver2-http-frontend closed. The connection had 1 associated session(s).I1007 22:10:10.706935 56557 impala-hs2-server.cc:821] FetchResults(): query_id=764d4313dbc64e20:2831560c00000000 fetch_size=1024I1007 22:10:10.707320   325 TAcceptQueueServer.cpp:340] New connection to server hiveserver2-http-frontend from client <Host: 172.18.0.1 Port: 51588>I1007 22:10:10.707453 56535 query-state.cc:652] 764d4313dbc64e20:2831560c00000001] Instance completed. instance_id=764d4313dbc64e20:2831560c00000001 #in-flight=18 status=OKI1007 22:10:10.713026   329 TAcceptQueueServer.cpp:227] TAcceptQueueServer: hiveserver2-http-frontend started connection setup for client <Host: 172.18.0.1 Port: 51588>I1007 22:10:10.713054   329 TAcceptQueueServer.cpp:245] TAcceptQueueServer: hiveserver2-http-frontend finished connection setup for client <Host: 172.18.0.1 Port: 51588>I1007 22:10:10.714377   127 control-service.cc:142] e1495f928c2cd4f6:eeda82aa00000000] ExecQueryFInstances(): query_id=e1495f928c2cd4f6:eeda82aa00000000 coord=b04a12d76e27:22000 #instances=6I1007 22:10:10.715759 56527 coordinator.cc:375] 6d4cba4d2e8ccc42:66ce26a800000000] started execution on 3 backends for query_id=6d4cba4d2e8ccc42:66ce26a800000000I1007 22:10:10.717445 56560 query-state.cc:643] e1495f928c2cd4f6:eeda82aa00000000] Executing instance. instance_id=e1495f928c2cd4f6:eeda82aa00000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=2 #in-flight=19I1007 22:10:10.718142   127 coordinator.cc:745] Backend completed: host=1161aa24dbe8:22000 remaining=3 query_id=99454b2820335de0:0a0a549e00000000I1007 22:10:10.718163   127 coordinator-backend-state.cc:338] query_id=99454b2820335de0:0a0a549e00000000: first in-progress backend: 38e8ee0f58f6:22000I1007 22:10:10.720214 56557 impala-hs2-server.cc:844] FetchResults(): query_id=764d4313dbc64e20:2831560c00000000 #results=0 has_more=true
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                lv Lars Volker
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: