[LUCENE-2019] map unicode process-internal codepoints to replacement character

• Type: Improvement
• Status: Reopened
• Priority: Minor
• Resolution: Unresolved
• Component/s: core/index
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 29/Oct/09 18:18
• Updated: 28/Aug/22 12:12

Description

A spinoff from LUCENE-2016.

There are several process-internal codepoints in unicode, we should not store these in the index.
Instead they should be mapped to replacement character (U+FFFD), so they can be used process-internally.

An example of this is how Lucene Java currently uses U+FFFF process-internally, it can't be in the index or will cause problems.