[LUCENE-10405] MemoryIndex: Binary and Sorted doc values should not be added to a BytesRefHash

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.1
• Lucene Fields: New
• Assignee: Ignacio Vera
• Reporter: Ignacio Vera
• Created: 04/Feb/22 14:00
• Updated: 28/Aug/22 16:35
• Resolved: 07/Feb/22 06:36

Description

Currently when we add a binary or sorted doc value in the MemoryIndex, it will get stored in a BytesRefHash. This is not necessary as we only expect one doc value per document so they don't need to be deduped.

In addition  a BytesRefHash has a limit on term size (~32kb) which those doc values don't have in normal codecs. Therefore as it s a different behaviour it can be considered a bug. We should store those doc values as a plain byte[] (or BytesRef). 

Issue Links

• links to: GitHub Pull Request #647