[SOLR-17376] Direct buffer memory leak in Solr Java Client

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 9.6.1
• Component/s: clients - java, http2
• Labels: memory-bug

Environment

Java version: 21.0.3+9-LTS

Jetty 10.0.21

Windows Server 2019

 

• Assignee: Unassigned
• Reporter: Alexander Veit
• Created: 23/Jul/24 12:43
• Updated: 25/Sep/24 19:27

Description

We use the Solr Java client in a log-running service to update documents in an index. As it turns out the Solr client allocates direct buffers that accumulate and never get released until JVM restart. After some hundreds of thousands instances of java.nio.DirectByteBuffer haven been created the system gets stuck and OutOfMemoryErrors occur inside the JVM.

Unfortunately the JVM does not terminate itself, probably because this is not an issue of Java heap memory. Therefore the service could not be restarted automatically but requires user interaction.

Although we have a rather complex scenario, we can almost certainly rule out that the leak is due to missing close() calls on the SolrClient instances within the JVM. SolrClients are generally reused during the lifetime of the JVM.

The Solr client was created like in the following code

SolrClient client = new Http2SolrClient.Builder("my-url").build();

 

The Solr requests were made like in the following code

QueryRequest request = new QueryRequest(params);
 
request.process(client, "my-collection");

A heap dump taken from an affected JVM showed the following instance counts:

org.apache.solr.client.solrj.impl.Http2SolrClient        2
org.eclipse.jetty.io.LogarithmicArrayByteBufferPool      1
org.eclipse.jetty.io.MappedByteBufferPool                2
java.nio.DirectByteBuffer                           458485

 

The DirectByteBuffers seem to be held alive by a thread-local map.

 

Issue Links

• links to: Jetty #12084