[LUCENE-2749] Co-occurrence filter

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Steven Rowe
• Created: 08/Nov/10 21:27
• Updated: 28/Aug/22 12:35

Description

The co-occurrence filter to be developed here will output sets of tokens that co-occur within a given window onto a token stream.

These token sets can be ordered either lexically (to allow order-independent matching/counting) or positionally (e.g. sliding windows of positionally ordered co-occurring terms that include all terms in the window are called n-grams or shingles).

The parameters to this filter will be:

• window size: this can be a fixed sequence length, sentence/paragraph context (these will require sentence/paragraph segmentation, which is not in Lucene yet), or over the entire token stream (full field width)
• minimum number of co-occurring terms: >= 2
• maximum number of co-occurring terms: <= window size
• token set ordering (lexical or positional)

One use case for co-occurring token sets is as candidates for collocations.

Issue Links

• relates to: LUCENE-5318 Co-occurrence counts from Concordance