[DRILL-5223] Drill should ensure balanced workload assignment at node level in order to get better query performance

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Query Planning & Optimization
• Labels: ready-to-commit
• Assignee: Padma Penumarthy
• Reporter: Jinfeng Ni
• Reviewer: Jinfeng Ni
• Created: 25/Jan/17 18:40
• Updated: 08/Feb/17 06:24
• Resolved: 08/Feb/17 06:24

Description

Drill's work assignment logic currently aims to achieve workload balance across different minor fragment (or slices) and honor data affinity in order to get as much local read as possible.

However, when the # of work units could not be evenly divided by # of minor fragments, the remaining work units would tender to go to the first subset of drill endpoints. This means the drill endpoints assigned with the remaining work units could have larger workload than the rest of them. When MuxExchange is enabled (by default), all the minor fragments on the same node have to send data to a single Muxer per node, and unbalanced workload assignment at node level could impact query elapse time. which is essentially decided by the slowest drill endpoint.

Some prototype experimental run shows that with more balanced workload assignment, Drill shows quite significant improvement for most of TPC-H queries.

Issue Links

• links to: GitHub Pull Request #730