XML-RPC
  1. XML-RPC
  2. XMLRPC-168

XML-RPC server deadlocks under heavy load

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.2
    • Fix Version/s: 3.1.3
    • Component/s: Source
    • Labels:
      None
    • Environment:
      Soalris

      Description

      When running a XML-RPC server under heavy load, it eventually deadlocks inside the thread pool that manages the 'worker' threads which handle the individual XML-RPC requests - the classes involved are org.apache.xmlrpc.util.ThreadPool and org.apache.xmlrpc.util.ThreadPool$Poolable. jstack on the hung process shows:

      ----------
      Found one Java-level deadlock:
      =============================
      "XML-RPC-13":
      waiting to lock monitor 0x08d10bec (object 0xbb6959c0, a org.apache.xmlrpc.util.ThreadPool),
      which is held by "XML-RPC Weblistener"
      "XML-RPC Weblistener":
      waiting to lock monitor 0x08d1186c (object 0xbd2ed340, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
      which is held by "XML-RPC-2"
      "XML-RPC-2":
      waiting to lock monitor 0x08d112f4 (object 0xbd2ed570, a org.apache.xmlrpc.util.ThreadPool$Poolable),
      which is held by "XML-RPC Weblistener"

      Java stack information for the threads listed above:
      ===================================================
      "XML-RPC-13":
      at org.apache.xmlrpc.util.ThreadPool.repool(Unknown Source)

      • waiting to lock <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
        at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source)
        "XML-RPC Weblistener":
        at org.apache.xmlrpc.util.ThreadPool$Poolable.start(Unknown Source)
      • waiting to lock <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
      • locked <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
        at org.apache.xmlrpc.util.ThreadPool.startTask(Unknown Source)
      • locked <0xbb6959c0> (a org.apache.xmlrpc.util.ThreadPool)
        at org.apache.xmlrpc.webserver.WebServer.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:619)
        "XML-RPC-2":
        at org.apache.xmlrpc.util.ThreadPool$Poolable.isShuttingDown(Unknown Source)
      • waiting to lock <0xbd2ed570> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
        at org.apache.xmlrpc.util.ThreadPool$Poolable.access$000(Unknown Source)
        at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(Unknown Source)
      • locked <0xbd2ed340> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)

      Found 1 deadlock.
      ----------

      And another slight variant on the above, from a debug build of revision 769436.

      ----------
      Found one Java-level deadlock:
      =============================
      "XML-RPC-6":
      waiting to lock monitor 0x0870d8ec (object 0xbac020f8, a org.apache.xmlrpc.util.ThreadPool),
      which is held by "XML-RPC Weblistener"
      "XML-RPC Weblistener":
      waiting to lock monitor 0x0814de4c (object 0xbad73820, a org.apache.xmlrpc.util.ThreadPool$Poolable$1),
      which is held by "XML-RPC-5"
      "XML-RPC-5":
      waiting to lock monitor 0x0814eacc (object 0xbad73b48, a org.apache.xmlrpc.util.ThreadPool$Poolable),
      which is held by "XML-RPC Weblistener"

      Java stack information for the threads listed above:
      ===================================================
      "XML-RPC-6":
      at org.apache.xmlrpc.util.ThreadPool.repool(ThreadPool.java:136)

      • waiting to lock <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
        at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:70)
        "XML-RPC Weblistener":
        at org.apache.xmlrpc.util.ThreadPool$Poolable.start(ThreadPool.java:106)
      • waiting to lock <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)
      • locked <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
        at org.apache.xmlrpc.util.ThreadPool.startTask(ThreadPool.java:168)
      • locked <0xbac020f8> (a org.apache.xmlrpc.util.ThreadPool)
        at org.apache.xmlrpc.webserver.WebServer.run(WebServer.java:338)
        at java.lang.Thread.run(Thread.java:619)
        "XML-RPC-5":
        at org.apache.xmlrpc.util.ThreadPool$Poolable.getTask(ThreadPool.java:99)
      • waiting to lock <0xbad73b48> (a org.apache.xmlrpc.util.ThreadPool$Poolable)
        at org.apache.xmlrpc.util.ThreadPool$Poolable.access$100(ThreadPool.java:47)
        at org.apache.xmlrpc.util.ThreadPool$Poolable$1.run(ThreadPool.java:59)
      • locked <0xbad73820> (a org.apache.xmlrpc.util.ThreadPool$Poolable$1)

      Found 1 deadlock.
      ----------

      1. ThreadPool.java
        8 kB
        Jochen Wiedmann
      2. Server.java
        4 kB
        Alan Burlison
      3. Client.java
        1 kB
        Alan Burlison
      4. Client.java
        2 kB
        Jochen Wiedmann
      5. ThreadPool.java
        7 kB
        Jochen Wiedmann

        Issue Links

          Activity

            People

            • Assignee:
              Jochen Wiedmann
              Reporter:
              Alan Burlison
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development