[ZEPPELIN-271] RemoteInterpreterServer deadlock

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.5.5
• Assignee: zenglinxi
• Reporter: zenglinxi
• Created: 30/Aug/15 03:18
• Updated: 02/Sep/16 08:27
• Resolved: 03/Sep/15 03:12

Description

After using zeppelin server for some days, we often found that paragraph can't run while the zeppelin server process is on, restart interpreter is just not work. Then we find out that the RemoteInterpreterServer is in deadlock, and can't be stopped by zeppelin server.
Unfortunately, this deadlocked RemoteInterpreterServer will still hold the hadoop resources, we need to kill the process manually each time.
We use jstack to monitor the RemoteInterpreterServer process, the key information is like that:

Found one Java-level deadlock:
=============================
"pool-1-thread-8":
waiting to lock monitor 0x00007f58a8006178 (object 0x00000000ebb9d298, a org.apache.zeppelin.interpreter.InterpreterGroup),
which is held by "pool-1-thread-4"
"pool-1-thread-4":
waiting to lock monitor 0x00007f58a8002f48 (object 0x00000000ecbd8008, a org.apache.zeppelin.interpreter.ClassloaderInterpreter),
which is held by "pool-1-thread-2"
"pool-1-thread-2":
waiting to lock monitor 0x00007f58a8006178 (object 0x00000000ebb9d298, a org.apache.zeppelin.interpreter.InterpreterGroup),
which is held by "pool-1-thread-4"