[IMPALA-2424] Rack-aware scheduling

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.2.4
• Component/s: Distributed Exec
• Labels: scalability scheduler
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Marcel Kinard
• Created: 25/Sep/15 18:33
• Updated: 19/Oct/21 11:16

Description

Currently, Impala makes an effort to schedule plan fragments local to the data that is being scanned; when no collocated impalad is available, the plan fragment is placed randomly.

In order to support configurations where Impala is run on a subset of the nodes in a cluster, we should schedule fragments within the same rack that holds the assigned scan ranges (if a collocated impalad isn't available).

See https://issues.apache.org/jira/browse/HADOOP-692 for details of how rack locality is recorded in hdfs.

Issue Links

• relates to: IMPALA-6088 Rack aware broadcast operator