[CASSANDRA-12078] [SASI] Move skip_stop_words filter BEFORE stemming

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.8
• Component/s: Feature/SASI

Environment

Cassandra 3.7, Cassandra 3.8

• Severity: Normal
• Since Version: 3.7
• Assignee: DuyHai Doan
• Reporter: DuyHai Doan

Authors

DuyHai Doan

Reviewers

Pavel Yaskevich

• Created: 23/Jun/16 12:18
• Updated: 16/Apr/19 09:30
• Resolved: 26/Jun/16 08:54

Description

Right now, if skip stop words and stemming are enabled, SASI will put stemming in the filter pipeline BEFORE skip_stop_words:

    private FilterPipelineTask getFilterPipeline()
    {
        FilterPipelineBuilder builder = new FilterPipelineBuilder(new BasicResultFilters.NoOperation());
     ...
        if (options.shouldStemTerms())
            builder = builder.add("term_stemming", new StemmingFilters.DefaultStemmingFilter(options.getLocale()));
        if (options.shouldIgnoreStopTerms())
            builder = builder.add("skip_stop_words", new StopWordFilters.DefaultStopWordFilter(options.getLocale()));
        return builder.build();
    }

The problem is that stemming before removing stop words can yield wrong results.

I have an example:

SELECT * FROM music.albums WHERE country='France' AND title LIKE 'danse' ALLOW FILTERING;

Because of stemming danse ( dance in English) becomes dans (the final vowel is removed). Then skip stop words is applied. Unfortunately dans (in in English) is a stop word in French so it is removed completely.

In the end the query is equivalent to SELECT * FROM music.albums WHERE country='France' and of course the results are wrong.

Attached is a trivial patch to move the skip_stop_words filter BEFORE stemming filter

/cc xedin jrwest beobal