[IMPALA-5200] Profile timers not updated during long-running sort

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: observability ramp-up supportability
• Epic Link: Improve Impala Profile User Experience
• Target Version: Product Backlog
• Epic Color: ghx-label-7
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 12/Apr/17 15:04
• Updated: 19/Dec/18 19:20

Description

If you have a query plan with a long-running sort operation (e.g. minutes), the profile timers are not updated to reflect the time spent in the sort until the sort starts returning rows.

E.g. this is a summary from a sort query that was running for a few hours . The summary was misleading and the "heat map" plan in the debug web UI is misleading - it showed the join as the "hot" operator. It would be ideal if we could somehow at least periodically update the time spent in the operator.

Operator              #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                   
----------------------------------------------------------------------------------------------------------------------------
05:MERGING-EXCHANGE        1    0.000ns    0.000ns        0     635.58M          0        -1.00 B  UNPARTITIONED            
03:SORT                    1    0.000ns    0.000ns        0     635.58M   47.86 GB      800.00 MB                           
02:HASH JOIN               1    4s859ms    4s859ms  771.02M     635.58M  162.11 MB       16.03 MB  INNER JOIN, BROADCAST    
|--04:EXCHANGE             1   38.988ms   38.988ms  247.20K     247.20K          0              0  BROADCAST                
|  01:SCAN HDFS            1    8s089ms    8s089ms  247.20K     247.20K    3.79 MB       32.00 MB  product b     
00:SCAN HDFS               1  209.997ms  209.997ms   15.09M     635.58M  185.27 MB      176.00 MB  sales a

http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-join-query-running-slow

Issue Links

• is related to: IMPALA-7318 Make cpu time limit a supported option