[SOLR-9454] Reduce object allocation during indexing because of JavaBinCodec.writeExternString()

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 6.7, 7.0
• Component/s: SolrCloud, update
• Labels: difficulty-medium impact-medium
• Assignee: Unassigned
• Reporter: Shalin Shekhar Mangar
• Created: 29/Aug/16 15:49
• Updated: 08/Jun/19 15:37

Description

I setup Java Flight Recorder to profile indexing a 650MB JSON file using bin/post on 2 shard, 2 replica setup. It shows that the JavaBinCodec.writeExternString(String) method contributes a lot of garbage during indexing in SolrCloud. More specifically, it contributes ~1GB of HashMap$Node objects and ~450MB of HashMap$Node[] objects.

Most of this allocation is because every request is serialized using a new instance of JavaBinUpdateRequestCodec which internally allocates a new HashMap for storing the extern strings.

We should explore keeping a global extern string map to eliminate redundant allocations.