[SOLR-16265] reduce memory usage of ContentWriter based requests in Http2SolrClient

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.4
• Assignee: Alex Deparvu
• Reporter: Chris M. Hostetter
• Created: 24/Jun/22 22:14
• Updated: 15/Oct/23 19:57
• Resolved: 08/Aug/23 16:56

Description

I recently noticed the code below exists in Http2SolrClient.createRequest...

if (contentWriter != null) {
  Request req = httpClient.newRequest(url + wparams.toQueryString()).method(method);
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  contentWriter.write(baos);

  // TODO reduce memory usage
  return req.content(
      new BytesContentProvider(contentWriter.getContentType(), baos.toByteArray()));

• AFAICT there is no (other) existing jira discussing this TODO
• This method is called for most "simple" HTTP2 based requests
  • Http2SolrClient or CloudHttp2SolrClient – but not ConcurrentUpdateHttp2SolrClient
• This block triggers for anything with a ContentWriter
  • ie: all UpdateRequests ... and in theory other custom requests
• Part of the issue seems to be that this code repurposes the ContentWriter "push" style API into a "pull" style Jetty client API
  • Even though Http2SolrClient has other code used only by ConcurrentUpdateHttp2SolrClient (initOutStream(...)) which does leverage a "push" style Jetty client API: OutputStreamContentProvider
• But more silly: we make one (serialized) byte[] of the data in memory inside the ByteArrayOutputStream then we call toByteArray() which makes a second copy of the byte[].

Issue Links

• links to: GitHub Pull Request #1789
• links to: GitHub Pull Request #1860

Sub-Tasks

• Http2SolrClient: Replace ByteArrayOutputStream.toByteArray() w/BAOS.getbuf(): Closed