[LUCENE-4127] negative offsets/deltas corrumption

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0-ALPHA
• Fix Version/s: 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 10/Jun/12 12:21
• Updated: 10/May/13 10:40
• Resolved: 03/Aug/12 13:33

Description

If offsets go negative or backwards, it can corrupt the index with DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS: the offsets will have wrong values (different from the term vectors) or even crazy values like -2147483645

The problem with this is that its not just theoretical: its too easy to do this with lucene's own analyzer chains (e.g. ngramtokenizer).

See issues such as LUCENE-3920 and some discussion on LUCENE-3738

The question is how to fix this, e.g. should we:

1. start enforcing that offsets cannot be crazy values in OffsetAttributeImpl/IndexWriter and fix the broken analyzers
2. leave offsets as a pair of opaque integers, declaring this a limitation of the current codec, and either workaround or throw UOE from the postings writer.