[IMPALA-4064] impala query is spent long time for ScannerThreadsTotalWallClockTime

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: Impala 2.3.0
• Component/s: Perf Investigation
• Labels: performance
• Environment: CDH 5.7.2-impala 2.3
• Assignee: Unassigned
• Reporter: hongchao
• Created: 02/Sep/16 09:12
• Updated: 13/Jun/17 05:21
• Resolved: 13/Jun/17 05:21

Description

In our CDH 5.5.2 with impala 2.3, we often met performance issue, the same queries in the same cluster, sometimes is fast, some times is very slow, our sql is very simple with only one aggregation(for detail pls check the query profile info) like bellow:
```sql
select Sum(c1), sum(c2)
FROM xxx
WHERE day between 20160818 and 20160824
```

we also check the profile, compare the slow and quick for the same sql, we found that: the slow query is wasting time for ScannerThreadsTotalWallClockTime, and only one node is extremely slow, others are quick. we also found the spent time is mostly wasted in InactiveTotalTime, it seems the cost is wasted on hdfs scan, but the all other executor node is not met the problem, we also checked the namenode server resource(no resource tension, include memory,cpu,network), the attached is the quick and slow profile data for the same query, the query only used less than 8M memory for each executor.

pls help us, tell me what exactly cost for ScannerThreadsTotalWallClockTime, how can we improve it? how can we make query stable and quickly, thx very much.