Negative wildcard searches on MultiSearcher not eliminating correctly.

• Key: LUCENE-1300
• Type: Bug
• Status: Resolved
• Resolution: Duplicate
• Priority: Minor
• Assignee: Unassigned
• Reporter: steve halsey
• Component/s: Search
• Affects Version/s: 2.1, 2.3, 2.3.1

Description

If you do a search for a negative wildcard query on a MultiSearcher where one of the searchers is empty e.g. "lucene -bug*" the hits returned incorrectly include articles with words that should be eliminated, e.g. "bug" and "bugs". This is because the query expansion is done on the index with docs in and the empty ndex separately and then combined as an OR to be run on the MultiSearcher. This incorrectly lets in docs that have the excluded wildcard terms, e.g. "bug" nd bugs". This bug would also show up with two indexes full of docs, and I can send a test to show that if required, but I think this test demonstrates the bug in the implest way.

The attached class TestMultiSearcherNegativeWildcardQueryExpansion.java can be put in with other tests in org.apache.lucene.search and run and will fail, showing the bug exists.

I have tested this bug with the currently unreleased 2.3.2 and the released 2.1 and 2.3.1 and it fails on all.

With lucene-1.5-rc1 it passes (with mods to make it work with old API) see TestMultiSearcherNegativeWildcardQueryExpansionWorksWith151.java attachment.

Comments

[steve halsey]

This test fails, demonstrating the existence of the negative wildcard query on MultiSearchers bug.

[steve halsey]

This test class shows the test working with the old 1.5.1 version of lucene, modified to make it work with that old API.

[Mark Miller]

Great catch Steve. The combine method in Query appears to be very flawed when it comes to MUST_NOT occurrences and truncation queries. Nasty little bug that does indeed appear to go back to 05. Thanks for all of the detailed info. I am sure someone will be right on top of this.

[Mark Miller]

Looks like the test worked before because things were even (it would appear) worse - the bug was that the multi term query was only expanded on the first index and then the resulting query was used on all of the indexes. The issue introducing the bug you have found was an attempt to fix this by expanding on each Reader and then attempting to make a single query that works across each Reader. The strategy seems to work in non MUST_NOT cases, but the query generated can just be wrong with a MUST_NOT occurrence. As you point out, the second index doesn't even have to be empty, and the second -() clause does not even have to be empty either - the generated query can still be wrong.

I don't see the obvious fix - somehow we need a query that expands against all of the subreaders as if one reader, or the combine method has to figure this out...neither seem easy to me...

[Mark Miller]

One option may be to do this:

Figure out how we can change Searchable (deprecation, whatever) and add a getIndexReader method. Make the getIndexReader method on MultiSearcher return a MultiReader with the underlying searchable Readers. Now on MultiSearcher rewrite, the query can be rewritten on a temp IndexSearcher that uses the MultiReader.

Ive tried it quickly, but I havn't thought out all the ramifications. I wouldnt be surprised if there was some biggies (being able to get an IndexReader off a Searchable would be quite the change)...and obviously changing Searchable pretty much sucks. Any other ideas though? I can't think of a way to make the combine method work right otherwise without recursing down the query and doing some really nasty bookkeeping stuff.

[Mark Miller]

Okay, I clearly underestimated the difficulties of this due to RemoteSearchable. I don't see how it can be done in any efficient manner when you have to work off a Searchable, and obviously you can't do anything with multiple Readers using the MultiSearcher on the client side, so it would seem making the Query.combine method work is the only option...except that doing such a thing would be really nasty I think.

[Mark Miller]

This is actually a dupe of an older issue.

There is no clean way to fix it with the current Searchable API. Avoid MultiSearcher if you can <g>

[steve halsey]

Hi Mark,

OK. Thanks for that. It is a rarely seen problem, because most times
when people want to eliminate a word e.g. lucen* then all of the terms
will be in both halves of the index and so the query expansion will be
acurate and the query will work.

Cheers

steve