[DRILL-3306] Concurrently Running hive queries results in a deadlock situation - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.1.0
Component/s: Storage - Hive
Labels:
None

Description

I tried submitting ~20 queries using 10 concurrent threads. Now the drillbit goes into an unresponsive state. Now even trying to launch a new sqlline session hangs (until hive.metastore.client.socket.timeout expires)

Below is the jstack output

Attaching to process ID 4662, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
Deadlock Detection:

java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
	at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
	... 15 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
	at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
	... 12 more

Jstack Output of the hive metastore

Attaching to process ID 32320, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
Deadlock Detection:

java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
	at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
	... 15 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
	at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
	... 12 more

Observing from the web ui, after every 10 mins one query changes state from pending to running. All the queries in the running state did not complete and I stopped the drillbit after waiting for 30 mins

Also at this stage attempting to stop the drillbit does not succeed and the script internally call the kill command on the process

I attached the contents of drillbit.log file. Let me know if you need anything else

Concurrently Running hive queries results in a deadlock situation

Details

Description

Attachments

Attachments

Activity

People

Dates