[IMPALA-7095] Improve scanner thread counters in HDFS and Kudu scans

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: observability
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 30/May/18 23:19
• Updated: 01/Apr/19 04:18
• Resolved: 11/Jul/18 23:22

Description

There are a few deficiencies here:

• We don't track the peak number of scanner threads. Consumers of the profile often confuse NumScannerThreadsStarted with the peak.
• Kudu scans are missing some metrics, e.g. AverageScannerThreadConcurrency. We should make sure that Kudu and HDFS are consistent.

We should clean this up, and maybe refactor the code so that less logic is duplicated