[IMPALA-4037] ChildQuery::Cancel() appears to violate lock ordering

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.5.5, Impala 2.8.0
• Component/s: Distributed Exec
• Labels: hang
• Target Version: Impala 2.8.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 29/Aug/16 16:14
• Updated: 14/Nov/16 09:58
• Resolved: 22/Sep/16 16:10

Description

The lock order is ImpalaServer::query_exec_state_map_lock_ -> QueryExecState::lock_, but there's a possible callstack that violates this.

• ImpalaServer::CancelInternal() holding QueryExecState::lock_
• calls QueryExecState::Cancel()
• calls ChildQuery::Cancel()
• calls ImpalaServer::CancelOperation()
• calls ImpalaServer::GetQueryExecState(), which grabs ImpalaServer::query_exec_state_map_lock_

This could lead to deadlock if another thread called GetQueryExecState() on the same ImpalaServer instance (i.e. HS2/beewax) for the same QueryExecState at the same time.

Issue Links

• is related to: IMPALA-4038 RPC delays for single query can lead to ImpalaServer not making progress on any queries