-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 4.0-ALPHA
-
Fix Version/s: 4.0-ALPHA
-
Component/s: core/index
-
Labels:None
-
Lucene Fields:New
If offsets go negative or backwards, it can corrupt the index with DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS: the offsets will have wrong values (different from the term vectors) or even crazy values like -2147483645
The problem with this is that its not just theoretical: its too easy to do this with lucene's own analyzer chains (e.g. ngramtokenizer).
See issues such as LUCENE-3920 and some discussion on LUCENE-3738
The question is how to fix this, e.g. should we:
- start enforcing that offsets cannot be crazy values in OffsetAttributeImpl/IndexWriter and fix the broken analyzers
- leave offsets as a pair of opaque integers, declaring this a limitation of the current codec, and either workaround or throw UOE from the postings writer.