[IMPALA-7482] Deadlock with unknown lock holder in JVM in java.security.Provider.getService()

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 3.0
• Fix Version/s: Not Applicable
• Component/s: Backend
• Labels: hang
• Target Version: Impala 3.4.0
• Epic Color: ghx-label-6
• Assignee: Joe McDonnell
• Reporter: Tim Armstrong
• Created: 24/Aug/18 00:16
• Updated: 19/Feb/20 20:16
• Resolved: 19/Feb/20 20:16

Description

We've seen several instances of these mystery deadlocks in impalad's embedded JVM. The signature is a deadlock stemming from sun.security.provider.Sun being locked by an unknown owner.

Found one Java-level deadlock:
=============================
"Thread-24":
  waiting to lock monitor 0x0000000012364688 (object 0x000000008027ef30, a sun.security.provider.Sun),
  which is held by UNKNOWN_owner_addr=0x0000000014120800

If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck. If it happens in the Kudu client it also causes hangs.

Issue Links

• relates to: HADOOP-15538 Possible RPC deadlock in Client
• relates to: HDFS-14015 Improve error handling in hdfsThreadDestructor in native thread local storage
• links to: Forum thread about similar deadlock in Kudu client

Sub-Tasks

• Abort stuck impalad if JVM deadlock is detected: Resolved