[SOLR-7971] Reduce memory allocated by JavaBinCodec to encode large strings

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 5.4, 6.0
• Component/s: Response Writers, SolrCloud
• Assignee: Shalin Shekhar Mangar
• Reporter: Shalin Shekhar Mangar
• Created: 25/Aug/15 16:25
• Updated: 09/May/16 18:49
• Resolved: 03/Sep/15 20:53

Description

As discussed in SOLR-7927, we can reduce the buffer memory allocated by JavaBinCodec while writing large strings.

https://issues.apache.org/jira/browse/SOLR-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700420#comment-14700420

The maximum Unicode code point (as of Unicode 8 anyway) is U+10FFFF (http://www.unicode.org/glossary/#code_point). This is encoded in UTF-16 as surrogate pair \uDBFF\uDFFF, which takes up two Java chars, and is represented in UTF-8 as the 4-byte sequence F4 8F BF BF. This is likely where the mistaken 4-bytes-per-Java-char formulation came from: the maximum number of UTF-8 bytes required to represent a Unicode code point is 4.

The maximum Java char is \uFFFF, which is represented in UTF-8 as the 3-byte sequence EF BF BF.

So I think it's safe to switch to using 3 bytes per Java char (the unit of measurement returned by String.length()), like CompressingStoredFieldsWriter.writeField() does.