[IMPALA-8305] Generate JUnitXML symptom to detect crashes due to a DCHECK

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Epic Link: Generate Jenkins Symptoms for Common Problems
• Epic Color: ghx-label-5
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 12/Mar/19 22:26
• Updated: 18/Mar/19 17:12
• Resolved: 18/Mar/19 17:12

Description

When an impalad or other process hits a DCHECK, it outputs a FATAL log with information about the check that failed. For example:

Log file created at: 2019/02/17 00:27:43
Running on machine: impala-ec2-centos74-r4-4xlarge-ondemand-0a7e.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0217 00:27:43.730983 109290 query-state.cc:604] 154c7230f098d1a5:e75d0b400000000] Check failed: is_cancelled_.Load() == 1 (0 vs. 1)

Like all impalad crashes, this can show up as random tests failing. This FATAL log should generate a JUnitXML symptom containing the log output.

The check will need to navigate some exceptions. For example, some backend tests intentionally hit DCHECKs (e.g. buffer-pool-test and some others). Also, there is a custom cluster test that fails impalad startup and generates a FATAL log that is not a DCHECK.