Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3306

Concurrently Running hive queries results in a deadlock situation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.1.0
    • Storage - Hive
    • None

    Description

      I tried submitting ~20 queries using 10 concurrent threads. Now the drillbit goes into an unresponsive state. Now even trying to launch a new sqlline session hangs (until hive.metastore.client.socket.timeout expires)

      Below is the jstack output

      Attaching to process ID 4662, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 24.65-b04
      Deadlock Detection:
      
      java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
      	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      	at sun.tools.jstack.JStack.main(JStack.java:102)
      Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
      	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
      	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
      	... 15 more
      Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      Exception in thread "main" java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      	at sun.tools.jstack.JStack.main(JStack.java:102)
      Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
      	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      	... 6 more
      Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
      	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
      	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
      	... 12 more
      

      Jstack Output of the hive metastore

      Attaching to process ID 32320, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 24.65-b04
      Deadlock Detection:
      
      java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
      	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
      	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      	at sun.tools.jstack.JStack.main(JStack.java:102)
      Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
      	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
      	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
      	... 15 more
      Can't print deadlocks:Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      Exception in thread "main" java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      	at sun.tools.jstack.JStack.main(JStack.java:102)
      Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
      	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
      	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      	... 6 more
      Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
      	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
      	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
      	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
      	... 12 more
      

      Observing from the web ui, after every 10 mins one query changes state from pending to running. All the queries in the running state did not complete and I stopped the drillbit after waiting for 30 mins

      Also at this stage attempting to stop the drillbit does not succeed and the script internally call the kill command on the process

      I attached the contents of drillbit.log file. Let me know if you need anything else

      Attachments

        1. DRILL-3306-1.patch
          4 kB
          Venki Korukanti
        2. error.log
          244 kB
          Rahul Kumar Challapalli

        Activity

          People

            vkorukanti Venki Korukanti
            rkins Rahul Kumar Challapalli
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: