[IMPALA-2151] Join build threads are not reported to the QueryResourceManager

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Bug
• Affects Version/s: Impala 2.0.1
• Component/s: Backend
• Labels: llama query-lifecycle resource-management
• Target Version: Product Backlog
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 23/Jul/15 23:39
• Updated: 07/Dec/16 17:42
• Resolved: 07/Dec/16 17:42

Description

CPU intensive threads are supposed to be allocated by the ThreadResourceMgr to keep track of the total number of threads that may be allocated between queries on the host. However, when RM is enabled, threads need to be reported to the QueryResourceMgr so that additional VCPUs can be requested from Yarn.

The blocking join operator currently attempts to get a thread from the ThreadResourceMgr, but if it is allocated, it doesn't report it to the QueryResourceMgr. Unfortunately the fix is more involved than simply reporting the thread to the QRM because it may cause a child scan node to be starved of the thread that it is supposed to have, thus causing the query to hang when it can't start any scan threads. A quick fix may involve changing the way the scan node handles its first scanner thread, but we should consider rethinking the CPU thread management duality presented by the ThreadResourceMgr and the QueryResourceMgr.