[IMPALA-4411] Kudu inserts violate lock ordering and could deadlock

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Distributed Exec
• Labels: hang
• Target Version: Impala 2.8.0
• Assignee: Matthew Jacobs
• Reporter: Tim Armstrong
• Created: 31/Oct/16 23:16
• Updated: 03/Nov/16 22:19
• Resolved: 03/Nov/16 22:19

Description

"IMPALA-3788: Support for Kudu 'read-your-writes' consistency" introduced a violation of the lock ordering in ImpalaServer:

  unique_lock<mutex> l(lock_);
  end_time_ = TimestampValue::LocalTime();
  summary_profile_.AddInfoString("End Time", end_time().DebugString());
  summary_profile_.AddInfoString("Query State", PrintQueryState(query_state_));
  query_events_->MarkEvent("Unregister query");

  if (coord_.get() != NULL) {
    // Update latest observed Kudu timestamp stored in the session.
    uint64_t latest_kudu_ts = coord_->GetLatestKuduInsertTimestamp();
    if (latest_kudu_ts > 0) {
      VLOG_RPC << "Updating session latest observed Kudu timestamp: " << latest_kudu_ts;
      lock_guard<mutex> session_lock(session_->lock);
      session_->kudu_latest_observed_ts = std::max<uint64_t>(
          session_->kudu_latest_observed_ts, latest_kudu_ts);
    }

The required lock acquisition order is SessionState::lock, then QueryExecState::lock_, but the above code acquires them in the opposite order. See IMPALA-4409 for some info on the lock ordering and examples where the locks are acquired in the reverse order.