I traced this into the way the XMLRPC is set up.
Atlassian XMLRPC Binder sets up Apache XMLRPC Client using the Commons HTTP Client 3.1 transport. HTTP Client is using the default SimpleHttpConnectionManager. These are all hard coded.
Because the binder is not reused, a new XMLRPC client is set up for each call. The XMLRPC client calls releaseConnection on the HTTP Client, which leaves the physical connection open for reuse - but with nothing set up to reuse it or terminate it, it is left open until GC happens on the socket. This is often quick, but some connections make it into the more permanent area of heap.
Medium term, the XMLRPC should be replaced with JAX-RS.
Potential short term solutions:
- adjust the binder to use the JDK URL Connection instead of the HTTP Client
- try and adjust the HTTP Client to use a multi-threaded connection manager, and then reuse the binder/xmlrpc-client/http-client
- configure HTTP Client's background job to cleanup idle connections
This needs further investigation.