subscribed

since (unescaped, unquoted) whitespace characters is the syntax that QueryParser uses to indicate the transition between clauses in a BooleanQuery, changing this (either in QueryParser or in some new query parser) would require coming up with some new syntax. (or in the case of a special case query parser like the FieldQParser in Solr, eliminating the possibility of expressing multi-clause queries)

subscribed - Current client has index full of clothing - a search for "dress shoes" will return results containing womens' dresses and running shoes. That's not really acceptable.

There is somewhat of a workaround for this for defType=lucene. Just escape every whitespace with a slash. So instead of new dress shoes search for new\ dress\ shoes. Of course you lose the ability to use normal lucene syntax.

I was hoping that this workaround would also work for defType=dismax, but with or without the escaped whitespace, queries get interpreted the same, incorrect way. For instance, assume I have the following line in my synonyms.txt: dress shoes => dress_shoes. Further assume that I have a field experiment that gets analysed with synonyms. A search for new dress shoes (with or without escaped spaces) will be interpreted as

+((experiment:new)~0.01 (experiment:dress)~0.01 (experiment:shoes)~0.01) (experiment:"new dress_shoes"~3)~0.01

The first clause is manditory and contains independently analysed tokens, so this will only match documents that contain "dress", "new", or "shoes", but never "dress shoes" because analysis takes place as expected at index time.

My thought on the original issue is that most query parsers should accumulate adjacent terms without intervening operators as a "term list" (quoted phrases would be a second level of term list) and that there needs to be a "list" interface for query term analysis.

Rather than simply present a raw text stream for the sequence/list of terms, each term would be fed into the token stream with an attribute that indicates which source term it belongs to.

The synonym processor would see a clean flow of terms and do its processing, but would also need to associate an id with each term of a multi-term synonym phrase so that multiple multi-word synonym choices for the same input term(s) don't get mixed up (i.e., multiple tokens at the same position with no indication of which original synonym phrase they came from).

By having those ID's for each multi-term synonym phrase, the caller of the list analyzer could then recontruct the tree of "OR" expressions for the various multi-term synonym phrases.

(How's it going Jack) Interesting idea, though I really need to crack into the QueryParser and play around a little bit before I have a strong opinion myself.

Bulk move 4.4 issues to 4.5 and 5.0

Move issue to Lucene 4.9.

Is that resolved? Anyone working on it? Thanks

Initial patch against the classic Lucene QueryParser grammar only. Runs of whitespace-separated text with no operators are now sent as a single input to the analyzer, rather than first splitting on whitespace.

There are 10 failing tests in the queryparser module (didn't try anywhere else yet), mostly caused by the way getFieldQuery() produces a nested query when there are multiple terms. I'll look into how to address that.

Patch fixing up two problems:

1. Multiple whitespace-separated terms' TermQuery-s within a BooleanClause are now flattened directly into the output Query list, rather than inserting the BooleanClause.
2. MultiFieldQuery's getFieldQuery() is modified to recombine multiple terms from each field's query, to produce a series of disjunctions of term against each field.

All queryparser module tests now pass, with the exception of the flexible query parser's TestStandardQP run with QueryParserTestBase.testQPA(). Since this patch doesn't modify anything about the flexible query parser, this is not surprising.

Patch, I think it's ready.

I've pulled MockSynonymFilter/Analyzer out into their own files in lucene-test-framework, and added tests for it, and added a fixed multi-term source synonym with one single-term target.

I added tests to TestQueryParser using the modified MockSynonymAnalyzer ensuring operators block multi-term analysis when they should and don't when they shouldn't.

I'll go make issues now for converting Solr's clone of this QueryParser, and the standard flexible query parser, to add the same capabilities.

+1. I did not think it would ever get fixed!

Incredible!

Strong +1

This one was really painful problem (unexpected "tokenization" by query parser!)
Thank you for fixing that!

Steve Rowe you are about to become everyone's hero and a household name!
Is this going to be in the upcoming 6.1?

No, it will not be in 6.1, but I expect it will make 6.2.

Patch adding an option to preserve the old behavior: {set/get}SplitOnWhitespace(), defaulting to true (the current behavior).

Though nobody said so here, on the Solr issue (SOLR-9185), a couple people mentioned that the old behavior should be preserved, and not be the default until a major release.

That's what this patch does.

Patch adds lucene-test-framework files missing from last version of the patch. Also adds a CHANGES entry.

I plan on committing in a couple days if there are no objections.

Okay, really final patch. On SOLR-9185 I was having trouble integrating the Solr standard QP's comment support with the whitespace tokenization I introduced here, so I tried switching the Solr parser back to ignoring both whitespace and comments, and it worked. The patch brings this grammar simplification back here too - in addition to many fewer whitespace mentions in the rules, fewer (and less complicated) lookaheads are required.

I've included the generated files in the patch.

No tests changed from the last patch.

All Lucene tests pass, and precommit passes.

Commit 7d092fac4eabab42006e8e2b5c8a149cb266350c in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7d092fa ]

LUCENE-2605: Add classic QueryParser option setSplitOnWhitespace() to control whether to split on whitespace prior to text analysis. Default behavior remains unchanged: split-on-whitespace=true.

Commit 17d113dac1e6081a48144679234b00a556210160 in lucene-solr's branch refs/heads/master from Steve Rowe
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=17d113d ]

LUCENE-2605: Add classic QueryParser option setSplitOnWhitespace() to control whether to split on whitespace prior to text analysis. Default behavior remains unchanged: split-on-whitespace=true.

Patch for master only that switches the default split-on-whitespace behavior from do to don't.

Committing shortly.

Commit 24e5c1c20581f115a2948a72a4cef82a232bb28d in lucene-solr's branch refs/heads/master from Steve Rowe
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=24e5c1c ]

LUCENE-2605: Classic QueryParser no longer splits on whitespace by default.

Removing the unused imports from QueryParserTokenManager.java makes precommit pass again.

Commit 032e31aea7ce5ea4d396302a1cbeab8a6c525f7d in lucene-solr's branch refs/heads/master from Varun Thacker
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=032e31a ]

LUCENE-2605: Remove unused imports

Removing the unused imports from QueryParserTokenManager.java makes precommit pass again.

Thanks Varun Thacker! I forgot to run precommit after I regenerated on master to change the default split-on-whitespace option.

Just adding a little note that SimpleQParser parses on whitespace and analyzes tokens separately. Some day it'd be nice to see that one updated.

David Smiley I think you can already disable splitting on whitespace with the SimpleQueryParser by disabling the whitespace operator?

Adrien Grand Indeed you are right! I didn't think to look in the operator list, and didn't notice a test for it. I just tried it out using Solr.

Bulk close resolved issues after 6.2.0 release.

Hi, I just installed Solr 6.2.1 to see if this patch fixes the query-time analyzer filter for multi-word synonyms.
But unfortunately I can see no difference: still it seems that the queryparser sends each whitespace separated term.
Reading the discussion above, it seems that splitting on whitespace is disabled by default now.

How can I test if this works?
Any example configuration would be very helpful!

Hi Dion Olsthoorn

Reading the discussion above, it seems that splitting on whitespace is disabled by default now.

No, that's wrong - from the commit messages (which you can see in comments on this issue above):

Default behavior remains unchanged: split-on-whitespace=true

In Lucene 7.0, though, the default before will switch to split-on-whitespace=false

How can I test if this works?
Any example configuration would be very helpful!

Have you tried setting setSplitOnWhitespace(false) on the query parser before using it?

Thanks for quick response, Steve. I overlooked your last patch was for master only, not for 6.x.
I'd like to try out the split-on-whitespace=false without coding, but by changing a configuration in solrconfig.xml.

Is that possible?

I'd like to try out the split-on-whitespace=false without coding, but by changing a configuration in solrconfig.xml.

Is that possible?

Not yet - Solr's "Lucene" aka standard query parser is a fork of the classic Lucene query parser that was modified on this issue to enable not splitting on whitespace - Solr doesn't have this support yet. There is a separate issue to apply the same changes to Solr here: SOLR-9185. I'm working on it.

What about SimpleQueryParser? Can it also be extended with setSplitOnWhitespace?

What about SimpleQueryParser? Can it also be extended with setSplitOnWhitespace?

SimpleQueryParser already has its own form of disabling whitespace splitting: just disable the WHITESPACE_OPERATOR, via the 3-arg ctor: http://lucene.apache.org/core/6_5_0/queryparser/org/apache/lucene/queryparser/simple/SimpleQueryParser.html#SimpleQueryParser-org.apache.lucene.analysis.Analyzer-java.util.Map-int-.

E.g. to disable only the WHITESPACE_OPERATOR and leave all other operators enabled (caveat: untested):

new SimpleQueryParser(analyzer, Collections.singletonMap("myfield", 1f), ~SimpleQueryParser.WHITESPACE_OPERATOR);