[IMPALA-7857] Log more information about statestore failure detector

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.2.0
• Component/s: Distributed Exec
• Labels: statestore supportability
• Target Version: Impala 3.2.0
• Epic Color: ghx-label-9
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 15/Nov/18 21:27
• Updated: 19/Nov/18 23:07
• Resolved: 19/Nov/18 23:07

Description

For debugging heartbeat failures (or non-failures) it would be useful to log enough information to infer the current state of the failure detector from logs. Specifically:

• Upon a failure, we should log the number of consecutive failures according to the failure detector. And also maybe how many failures remain until it's considered to be failed.
• We should log when the failure count is reset to 0 by a successful heartbeat.

Currently if there are occasional failures it's hard to tell with certainty whether it was reset correctly.