[IMPALA-10578] Big Query influence other query seriously when hardware not reach limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: Impala 3.4.0
• Component/s: Backend
• Environment: impala-3.4 ; 80 machines with 96 cpu and 256GB mem ; scratch-dir is on separate disk different from HDFS data dir
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: wesleydeng_nouse
• Created: 10/Mar/21 12:03
• Updated: 11/Jun/21 07:29
• Resolved: 10/Jun/21 08:44

Description

When a big query is running(use mt_dop=8), other query is very difficult to start. 

A small query (select distinct one field from a small table)  may take about 1 minutes, normallly it take only about 1~3 second.

 From the impalad log, I found a incomprehensible log like this:

---------------

About the gap between "Handling call" and "Deserializing Batch", I found another path : 
--KrpcDataStreamRecvr::SenderQueue::AddBatch

  ----EnqueueDeferredRpc(move(payload), l);   // after dequeue, will call KrpcDataStreamRecvr::SenderQueue::AddBatchWork

--------------- 

 

 

When the Big query is running, data spilled  has happened because mem_limit was set and this big query waste a lot of memory.

 

In the attchment, I append the profile of big query and small query. The small query can be finished in seconds normally. the timeline of small query show  as below:

Query Timeline: 21m39s

• Query submitted: 48.846us (48.846us)
• Planning finished: 2.934ms (2.886ms)
• Submit for admission: 12.572ms (9.637ms)
• Completed admission: 13.622ms (1.050ms)
• Ready to start on 56 backends: 15.271ms (1.649ms)
  • All 56 execution backends (171 fragment instances) started: 18s505ms (18s489ms)*
• Rows available: 51s770ms (33s265ms)
• First row fetched: 57s220ms (5s449ms)
• Last row fetched: 59s119ms (1s899ms)
• Released admission control resources: 1m1s (2s223ms)
• AdmissionControlTimeSinceLastUpdate: 80.000ms
• ComputeScanRangeAssignmentTimer: 439.749us