[IMPALA-11275] Dump thread debug information when crashing/generating a minidump

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: Backend
• Epic Link: SupportabilityRoadMap
• Epic Color: ghx-label-4
• Assignee: Michael Smith
• Reporter: Joe McDonnell
• Created: 28/Apr/22 18:24
• Updated: 23/Aug/22 05:52
• Resolved: 13/May/22 21:14

Description

When Impala crashes and generates a minidump, it is useful to know what query the thread was executing when it crashed. There is already some information stored on the stack in the ThreadDebugInfo, but stack resolution does not always work (e.g. if in codegen code). It would be useful to dump the thread information to stdout/stderr when generating the minidump. This could accompany the "Wrote minidump to..." message generated here:

static bool DumpCallback(const google_breakpad::MinidumpDescriptor& descriptor,
    void* context, bool succeeded) {
  // See if a file was written successfully.
  if (succeeded) {
    // Write message to stdout/stderr, which will usually be captured in the INFO/ERROR
    // log.
    const char msg[] = "Wrote minidump to ";
    const int msg_len = sizeof(msg) / sizeof(msg[0]) - 1;
    const char* path = descriptor.path();
    // We use breakpad's reimplementation of strlen to avoid calling into libc.
    const int path_len = my_strlen(path);
    // We use the linux syscall support methods from chromium here as per the
    // recommendation of the breakpad docs to avoid calling into other shared libraries.
    sys_write(STDOUT_FILENO, msg, msg_len);
    sys_write(STDOUT_FILENO, path, path_len);
    sys_write(STDOUT_FILENO, "\n", 1);
    sys_write(STDERR_FILENO, msg, msg_len);
    sys_write(STDERR_FILENO, path, path_len);
    sys_write(STDERR_FILENO, "\n", 1);
  }
  // Return the value received in the call as described in the minidump documentation. If
  // this values is true, then no other handlers will be called. Breakpad will still crash
  // the process.
  return succeeded;
}

https://github.com/apache/impala/blob/master/be/src/util/minidump.cc#L73

 

Issue Links

• causes: IMPALA-11291 minidump-test is flaky