[IMPALA-2696] Simple scheduler always uses the same replica which can create hot spots

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.5.0
• Component/s: Backend
• Labels: performance ramp-up scalability scheduler
• Target Version: Impala 2.5.0
• Assignee: Lars Volker
• Reporter: Mostafa Mokhtar
• Created: 21/Nov/15 02:01
• Updated: 25/Mar/16 17:24
• Resolved: 03/Feb/16 02:51

Description

When SimpleScheduler::ComputeScanRangeAssignment is assigning locations to scan ranges the same replica will be used over and over again. This behavior is good for leveraging the buffer cache but it creates hot spots on the cluster if there are multiple users reading the same block.
An extreme usecase is a single block file being read by multiple concurrent users again and again, with today's scheduling a single node on the cluster will be hot with all the remaining nodes left idle.

The proposal is to round robin the replica location picked for each scan range so that the load is evened out, sine HDFS by default has 3 replicas the load should get distributed over 3 nodes.

The new behavior should be off by default and enabled via a flag.

Once Impala adds resource based scheduling we should change this behavior.