Lucene - Core
  1. Lucene - Core
  2. LUCENE-6249

StandardQueryParser doesn't support pure negative clauses

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.10.3, 5.0
    • Fix Version/s: 4.10.5, 5.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      At first I thought it's by design but Uwe says it's a bug. SQP emits this:

      StandardQueryParser qp = new StandardQueryParser();
      qp.setDefaultOperator(Operator.AND);
         
      System.out.println(qp.parse("summary:foo", "deffld"));
      System.out.println(qp.parse("-summary:foo", "deffld"));
      System.out.println(qp.parse("!summary:foo", "deffld"));
      System.out.println(qp.parse("NOT summary:foo", "deffld"));
      
      summary:foo
      summary:foo
      summary:foo
      summary:foo
      

        Issue Links

          Activity

          Hide
          Dawid Weiss added a comment - - edited

          The problem doesn't seem to be with the grammar, but with query building after the parse tree is acquired.

          The tree is fine:

          <modifier operation='MOD_NOT'>
          <field start='9' end='12' field='summary' text='foo'/>
          </modifier>
          

          but it eventually goes to a single ModifierQueryNodeBuilder which just returns the unmodified underlying query. I don't fully understand the parser, looking into it.

          Show
          Dawid Weiss added a comment - - edited The problem doesn't seem to be with the grammar, but with query building after the parse tree is acquired. The tree is fine: <modifier operation='MOD_NOT'> <field start='9' end='12' field='summary' text='foo'/> </modifier> but it eventually goes to a single ModifierQueryNodeBuilder which just returns the unmodified underlying query. I don't fully understand the parser, looking into it.
          Hide
          Dawid Weiss added a comment -

          Ok, it looks like any top-level ModifierQueryNode will be ignored because the only reason it exists is that it modifies the behavior of boolean nodes, etc.

          I think a top-level check and conversion into a Boolean node should be applied if modifier query node is the top-node.

          Show
          Dawid Weiss added a comment - Ok, it looks like any top-level ModifierQueryNode will be ignored because the only reason it exists is that it modifies the behavior of boolean nodes, etc. I think a top-level check and conversion into a Boolean node should be applied if modifier query node is the top-node.
          Hide
          Dawid Weiss added a comment -

          Here's a modification to the parser that I think handles this.

          Show
          Dawid Weiss added a comment - Here's a modification to the parser that I think handles this.
          Hide
          Dawid Weiss added a comment -

          Finished a full test run, all tests pass with this patch (Solr and Lucene). If nobody minds, I'll commit it in tomorrow.

          Show
          Dawid Weiss added a comment - Finished a full test run, all tests pass with this patch (Solr and Lucene). If nobody minds, I'll commit it in tomorrow.
          Hide
          ASF subversion and git services added a comment -

          Commit 1660315 from Dawid Weiss in branch 'dev/trunk'
          [ https://svn.apache.org/r1660315 ]

          LUCENE-6249: StandardQueryParser doesn't support pure negative clauses.
          LUCENE-6248: Remove unused odd constants from StandardSyntaxParser.jj

          Show
          ASF subversion and git services added a comment - Commit 1660315 from Dawid Weiss in branch 'dev/trunk' [ https://svn.apache.org/r1660315 ] LUCENE-6249 : StandardQueryParser doesn't support pure negative clauses. LUCENE-6248 : Remove unused odd constants from StandardSyntaxParser.jj
          Hide
          ASF subversion and git services added a comment -

          Commit 1660316 from Dawid Weiss in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1660316 ]

          LUCENE-6249: StandardQueryParser doesn't support pure negative clauses.
          LUCENE-6248: Remove unused odd constants from StandardSyntaxParser.jj

          Show
          ASF subversion and git services added a comment - Commit 1660316 from Dawid Weiss in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1660316 ] LUCENE-6249 : StandardQueryParser doesn't support pure negative clauses. LUCENE-6248 : Remove unused odd constants from StandardSyntaxParser.jj
          Hide
          Timothy Potter added a comment -

          Bulk close after 5.1 release

          Show
          Timothy Potter added a comment - Bulk close after 5.1 release
          Hide
          Jigar Shah added a comment -

          This issue was impacting queries where NOT operator is used within parenthesis which is fixed now,

          Example:

          lottery AND (NOT ticket)
          Lucene 4.10.3: +lottery +ticket [Incorrect]
          Lucene 5.2.0: +lottery +(-ticket) [Correct]

          Can we add one testcase having NOT with parenthesis?

          Show
          Jigar Shah added a comment - This issue was impacting queries where NOT operator is used within parenthesis which is fixed now, Example: lottery AND (NOT ticket) Lucene 4.10.3: +lottery +ticket [Incorrect] Lucene 5.2.0: +lottery +(-ticket) [Correct] Can we add one testcase having NOT with parenthesis?
          Hide
          Dawid Weiss added a comment -

          Please create w new issue and provide a respective patch there.

          Show
          Dawid Weiss added a comment - Please create w new issue and provide a respective patch there.
          Hide
          Jigar Shah added a comment -

          Provided patch in LUCENE-6857.

          I applied patch attached to the issue on 4.10.3 branch and it works!! Is it possible to get this fix in 4.10.x version of Lucene ?

          Show
          Jigar Shah added a comment - Provided patch in LUCENE-6857 . I applied patch attached to the issue on 4.10.3 branch and it works!! Is it possible to get this fix in 4.10.x version of Lucene ?
          Hide
          Dawid Weiss added a comment -

          To be honest I don't think version 4.x is even planned unless something major pops up. I think you should consider switching to 5.x line (or fork your patched version).

          Show
          Dawid Weiss added a comment - To be honest I don't think version 4.x is even planned unless something major pops up. I think you should consider switching to 5.x line (or fork your patched version).
          Hide
          Jigar Shah added a comment -

          It seems this fix is very important considering the following case, it returns hits when it should not return, to me it seems a major bug.

          lottery AND (NOT ticket)
          Lucene 4.10.3: +lottery +ticket [Incorrect] gives results just because its parsed incorrectly.
          Lucene 5.2.0: +lottery +(-ticket) [Correct]

          And this fix seems to be safe for 4.10 version !! I am not in hurry to get this, But it will be helpful in case some production systems are on 4.x version, and don't want to re-index there database for 5.x just for query parser change.

          Show
          Jigar Shah added a comment - It seems this fix is very important considering the following case, it returns hits when it should not return, to me it seems a major bug. lottery AND (NOT ticket) Lucene 4.10.3: +lottery +ticket [Incorrect] gives results just because its parsed incorrectly. Lucene 5.2.0: +lottery +(-ticket) [Correct] And this fix seems to be safe for 4.10 version !! I am not in hurry to get this, But it will be helpful in case some production systems are on 4.x version, and don't want to re-index there database for 5.x just for query parser change.
          Hide
          ASF subversion and git services added a comment -

          Commit 1711034 from Dawid Weiss in branch 'dev/branches/lucene_solr_4_10'
          [ https://svn.apache.org/r1711034 ]

          LUCENE-6249: StandardQueryParser doesn't support pure negative clauses.
          LUCENE-6248: Remove unused odd constants from StandardSyntaxParser.jj

          Show
          ASF subversion and git services added a comment - Commit 1711034 from Dawid Weiss in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1711034 ] LUCENE-6249 : StandardQueryParser doesn't support pure negative clauses. LUCENE-6248 : Remove unused odd constants from StandardSyntaxParser.jj
          Hide
          Dawid Weiss added a comment -

          Backport to 4.10.5

          Show
          Dawid Weiss added a comment - Backport to 4.10.5
          Hide
          Jigar Shah added a comment -

          Thanks for this Backport

          Show
          Jigar Shah added a comment - Thanks for this Backport
          Hide
          Dawid Weiss added a comment -

          You're welcome!

          Show
          Dawid Weiss added a comment - You're welcome!

            People

            • Assignee:
              Dawid Weiss
              Reporter:
              Dawid Weiss
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development