[IMPALA-1150] Impala should limit the per-query core estimate to Yarn's per-node limit

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Bug
• Affects Version/s: Impala 1.4
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: llama resource-management
• Target Version: Product Backlog
• Assignee: Matthew Jacobs
• Reporter: Henry Robinson
• Created: 12/Aug/14 01:58
• Updated: 07/Dec/16 17:46
• Resolved: 07/Dec/16 17:46

Description

Yarn has yarn.nodemanager.resource.cpu-vcores which sets the maximum number of cores a single node can allocate. Impala uses Java's availableCores() method which looks at the whole machine. When RM is enabled, we should limit the estimated number of cores per-node to yarn.nodemanager.resource.cpu-vcores.

This is probably easiest to do in the backend in QuerySchedule::GetPerHostVCores().