Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.8.0
Description
After running large scale concurrent tests it was obvious that a dedicated coordinator.
During the concurrency tests resource utilization in terms of CPU, Memory and Network were significantly higher than other worker nodes.
The increase in resource utilization was mostly due to
- Sending exec plan fragment to remote fragments, ExecPlanFragment can become very large for complex queries with lots of files & partitions
- Runtime filters consume lots of untracked memory when the bloom filters are aggregated
- Runtime filters consume lots of Network throughput on the coordinator as it receives, aggregates and sends the filters
- Aggregating RunProfile counters
- When CPU utilization on the coordinator is high remote fragments occasionally fail with "Couldn't get a client for impala-8-2.vpc.cloudera.com:22000 Reason: Couldn't open transport for impala-compete-8-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)"
- Slow start of remote query fragments specially when the coordinator node is under heavy load
- Admission control counters can become out of sync when using multiple coordinators
Metadata should naturally benefit from running a dedicated coordinator as metadata is less likely to become out of sync and one day we can get rid of expensive catalog updates.
The dedicated coordinator should only run the Coordinator Fragment and not participate in any scans, aggs etc..
This work will require CM support.
Running a dedicated coordinator significantly improved performance of complex TPC-DS queries
| Query | Baseline (seconds) | Dedicated coord (seconds) | Speedup % |
|---|---|---|---|
| 74 | 73 | 43 | 70% |
| 75 | 152 | 91 | 67% |
| 4 | 127 | 78 | 63% |
| 25 | 53 | 34 | 56% |
| 95 | 35 | 23 | 52% |
| 11 | 82 | 54 | 52% |
| 85 | 37 | 25 | 48% |
| 17 | 71 | 48 | 48% |
| 64 | 147 | 101 | 46% |
| 29 | 65 | 47 | 38% |
| 72 | 408 | 312 | 31% |
| 18 | 36 | 28 | 29% |
| 44 | 36 | 28 | 29% |
| 60 | 6 | 5 | 20% |
| 77 | 12 | 10 | 20% |
| 50 | 43 | 36 | 19% |
| 5 | 60 | 51 | 18% |
| 81 | 14 | 12 | 17% |
| 23 | 198 | 170 | 16% |
| 80 | 29 | 25 | 16% |
| 35 | 22 | 19 | 16% |
| 68 | 15 | 13 | 15% |
| 78 | 551 | 486 | 13% |
| 31 | 36 | 32 | 13% |