[IMPALA-1972] Queries that take a long time to plan can cause webserver to block other queries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: hang performance
• Target Version: Product Backlog
• Assignee: Bharath Vissapragada
• Reporter: Henry Robinson
• Created: 30/Apr/15 23:10
• Updated: 04/Sep/18 18:56
• Resolved: 24/May/17 15:45

Description

Summary

Trying to get the details of a query through the debug web page while the query is planning will block other queries (and the UI itself), because query_exec_state_map_lock_ will be held for the duration of planning.

Details

While a query is planning, it holds onto its query exec state's lock:

   lock_guard<mutex> l(*(*exec_state)->lock());

    // register exec state as early as possible so that queries that
    // take a long time to plan show up, and to handle incoming status
    // reports before execution starts.
    RETURN_IF_ERROR(RegisterQuery(session_state, *exec_state));
    *registered_exec_state = true;

    // GetExecRequest() does planning
    RETURN_IF_ERROR((*exec_state)->UpdateQueryStatus(
        exec_env_->frontend()->GetExecRequest(query_ctx, &result)));

Query details callback

ImpalaServer::QuerySummaryCallback, which handles /query_plan, tries to get the same exec state's lock (see true argument to GetQueryExecState().

shared_ptr<QueryExecState> exec_state = GetQueryExecState(query_id, true);

GetQueryExecState() holds query_exec_state_map_lock_ while it waits to get the QueryExecState's lock:

shared_ptr<ImpalaServer::QueryExecState> ImpalaServer::GetQueryExecState(
    const TUniqueId& query_id, bool lock) {
  lock_guard<mutex> l(query_exec_state_map_lock_);
  QueryExecStateMap::iterator i = query_exec_state_map_.find(query_id);
  if (i == query_exec_state_map_.end()) {
    return shared_ptr<QueryExecState>();
  } else {
    if (lock) i->second->lock()->lock();
    return i->second;
  }
}

So until planning is finished, no query can get query_exec_state_map_lock_, which it needs to execute.

What can we do?

In the short term, maybe we can add TryGetQueryExecState() which will indicate if the query exists but its lock can't be taken.

Or we might be able to let go of query_exec_state_map_lock_ as soon as we find the entry, and before taking its lock:

shared_ptr<ImpalaServer::QueryExecState> ImpalaServer::GetQueryExecState(
    const TUniqueId& query_id, bool lock) {
  shared_ptr<QueryExecState> ret;
  {
    lock_guard<mutex> l(query_exec_state_map_lock_);
    QueryExecStateMap::iterator i = query_exec_state_map_.find(query_id);
    if (i == query_exec_state_map_.end()) {
      return shared_ptr<QueryExecState>();
    } else {
      ret = i->second;
    }
  } // give up query_exec_state_map_lock_

  if (lock) ret->lock()->lock();
  return ret;
}

Issue Links

• breaks: IMPALA-5411 Excessive logging while queries are loading metadata from ImpalaServer::GetRuntimeProfileStr
• is duplicated by: IMPALA-7505 impalad webserver hang in getting a lock of QueryExecStatus
• is duplicated by: IMPALA-1970 impalad web UI becomes unresponsive in certain situations
• is related to: IMPALA-3882 Fix some locking / concurrency issues in ImpalaServer and QueryExecState