[IMPALA-5432] SetMemLimitExceeded DCHECK no longer valid

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Epic Color: ghx-label-7
• Assignee: John Sherman
• Reporter: John Sherman
• Created: 04/Jun/17 20:55
• Updated: 05/Jun/17 20:42
• Resolved: 05/Jun/17 20:42

Description

DCHECK(query_status_.IsMemLimitExceeded()) fails due to conditional setting of MemLimitExceeded status. The query_status is only set to MemLimitExceeded if the prior status was ok (query_status_.ok).

void RuntimeState::SetMemLimitExceeded(MemTracker* tracker,
    int64_t failed_allocation_size, const ErrorMsg* msg) {
  Status status = tracker->MemLimitExceeded(this, msg == nullptr ? "" : msg->msg(),
      failed_allocation_size);
  {
    lock_guard<SpinLock> l(query_status_lock_);
    if (query_status_.ok()) query_status_ = status;
  }
  LogError(status.msg());
  // Add warning about missing stats except for compute stats child queries.
  if (!query_ctx().__isset.parent_query_id &&
      query_ctx().__isset.tables_missing_stats &&
      !query_ctx().tables_missing_stats.empty()) {
    LogError(ErrorMsg(TErrorCode::GENERAL,
        GetTablesMissingStatsWarning(query_ctx().tables_missing_stats)));
  }
  DCHECK(query_status_.IsMemLimitExceeded());
}

This was discovered while running: ./tests/run-tests.py query_test/test_queries.py -k test_very_large_strings
which caused some impalad instances to hit the DCHECK and generate minidumps in my environment (and fail the test).

Removing the DCHECK still doesn't allow the test to pass due to what I think is better memory error reporting, but I'm still investigating the failures to better understand what is going on.