[IMPALA-10705] Expose query retry failure

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Labels: observability
• Epic Color: ghx-label-4
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 17/May/21 08:18
• Updated: 17/May/21 08:18

Description

When encountering failures in submitting the retried query, there are no means for clients to get the reason. To be specifit, in QueryDriver::RetryQueryFromThread(), we handle errors in this pattern:

  // Run the new query.
  status = retry_request_state->Exec();
  if (!status.ok()) {
    string error_msg =
        Substitute("Exec for new query with id $0 failed", PrintId(retry_query_id));
    HandleRetryFailure(&status, &error_msg, request_state, retry_query_id);
    return;
  }

HandleRetryFailure() tries to add the non-ok status to the query state:

void QueryDriver::HandleRetryFailure(Status* status, string* error_msg,
    ClientRequestState* request_state, const TUniqueId& retry_query_id) {
  DCHECK(status != nullptr && !status->ok());
  status->AddDetail(
      Substitute("Failed to retry query $0", PrintId(request_state->query_id())));
  status->AddDetail(*error_msg);
  discard_result(request_state->UpdateQueryStatus(*status));
  parent_server_->UnregisterQueryDiscardResult(retry_query_id, false, status);
}

But it's ignored since the original query already has a non-ok status (otherwise it won't fail):

Status ClientRequestState::UpdateQueryStatus(const Status& status) {
  // Preserve the first non-ok status
  if (!status.ok() && query_status_.ok()) {
    UpdateExecState(ExecState::ERROR);
    query_status_ = status;
    summary_profile_->AddInfoStringRedacted(QUERY_STATUS_KEY, query_status_.GetDetail());
  }

  return status;
}

So the failure is neither logged or shown in the query profile.