[XMLRPC-168] XML-RPC server deadlocks under heavy load

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.1.3
• Component/s: Source
• Environment: Soalris
• Assignee: Jochen Wiedmann
• Reporter: Alan Burlison
• Created: 28/Apr/09 11:29
• Updated: 02/May/09 11:52
• Resolved: 29/Apr/09 20:06

Description

When running a XML-RPC server under heavy load, it eventually deadlocks inside the thread pool that manages the 'worker' threads which handle the individual XML-RPC requests - the classes involved are org.apache.xmlrpc.util.ThreadPool and org.apache.xmlrpc.util.ThreadPool$Poolable. jstack on the hung process shows:

----------
Found one Java-level deadlock:
=============================
"XML-RPC-13":
waiting to lock monitor 0x08d10bec (object 0xbb6959c0, a org.apache.xmlrpc.util.ThreadPool),
which is held by "XML-RPC Weblistener"
"XML-RPC Weblistener":
waiting to lock monitor 0x08d1186c (object 0xbd2ed340, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
which is held by "XML-RPC-2"
"XML-RPC-2":
waiting to lock monitor 0x08d112f4 (object 0xbd2ed570, a org.apache.xmlrpc.util.ThreadPool$Poolable),
which is held by "XML-RPC Weblistener"

Java stack information for the threads listed above:
===================================================
"XML-RPC-13":
at org.apache.xmlrpc.util.ThreadPool.repool(Unknown Source)

• waiting to lock <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
  at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source)
  "XML-RPC Weblistener":
  at org.apache.xmlrpc.util.ThreadPool$Poolable.start(Unknown Source)
• waiting to lock <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
• locked <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
  at org.apache.xmlrpc.util.ThreadPool.startTask(Unknown Source)
• locked <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
  at org.apache.xmlrpc.webserver.WebServer.run(Unknown Source)
  at java.lang.Thread.run(Thread.java:619)
  "XML-RPC-2":
  at org.apache.xmlrpc.util.ThreadPool$Poolable.isShuttingDown(Unknown Source)
• waiting to lock <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
  at org.apache.xmlrpc.util.ThreadPool$Poolable.access$000(Unknown Source)
  at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source)
• locked <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)

Found 1 deadlock.
----------

And another slight variant on the above, from a debug build of revision 769436.

----------
Found one Java-level deadlock:
=============================
"XML-RPC-6":
waiting to lock monitor 0x0870d8ec (object 0xbac020f8, a org.apache.xmlrpc.util.ThreadPool),
which is held by "XML-RPC Weblistener"
"XML-RPC Weblistener":
waiting to lock monitor 0x0814de4c (object 0xbad73820, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
which is held by "XML-RPC-5"
"XML-RPC-5":
waiting to lock monitor 0x0814eacc (object 0xbad73b48, a org.apache.xmlrpc.util.ThreadPool$Poolable),
which is held by "XML-RPC Weblistener"

Java stack information for the threads listed above:
===================================================
"XML-RPC-6":
at org.apache.xmlrpc.util.ThreadPool.repool(ThreadPool.java:136)

• waiting to lock <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
  at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:70)
  "XML-RPC Weblistener":
  at org.apache.xmlrpc.util.ThreadPool$Poolable.start(ThreadPool.java:106)
• waiting to lock <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
• locked <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
  at org.apache.xmlrpc.util.ThreadPool.startTask(ThreadPool.java:168)
• locked <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
  at org.apache.xmlrpc.webserver.WebServer.run(WebServer.java:338)
  at java.lang.Thread.run(Thread.java:619)
  "XML-RPC-5":
  at org.apache.xmlrpc.util.ThreadPool$Poolable.getTask(ThreadPool.java:99)
• waiting to lock <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
  at org.apache.xmlrpc.util.ThreadPool$Poolable.access$100(ThreadPool.java:47)
  at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:59)
• locked <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)

Found 1 deadlock.
----------

Issue Links

• relates to: XMLRPC-167 XmlRpcClient is supposed to be thread-safe but it isn't