[IMPALA-7483] Abort stuck impalad if JVM deadlock is detected

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-3
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 24/Aug/18 00:24
• Updated: 27/Aug/18 15:44
• Resolved: 27/Aug/18 15:44

Description

tlipcon suggested this as a mitigation for IMPALA-7482 to get the cluster unstuck.

The JVM exposes some functionality to detect deadlock threads: https://docs.oracle.com/javase/7/docs/api/java/lang/management/ThreadMXBean.html#findDeadlockedThreads()

We could have a background thread try to detect java-level deadlocks.

This comment in the docs is slightly offputting "This method is designed for troubleshooting use, but not for synchronization control. It might be an expensive operation." We should probably understand what this means and what the consequences are before running this always.

One option is to only check for deadlocks if some operations except a performance threshold. The disadvantage of this is that we need to have thresholds for every operation that can deadlock.