[IMPALA-2564] Introduce mechanism to limit query fan-out

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.2
• Component/s: Distributed Exec
• Labels: customer performance scalability
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 17/Oct/15 00:08
• Updated: 28/Jun/18 21:27

Description

The target use case is small queries on large clusters.

Today Impala schedules queries on all Impalad instances regardless of how much data each Impalad would read, this results in spreading the work too thin between nodes and exposes undesired scalability issues.

The proposal is to introduce a parameter that controls the Min/Max amount of data read by a single Impala instance.
The SimpleScheduler would combine several splits together in order to satisfy the Min size requirements for a single Impalad before moving on the to the next node.