Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9786

Defer creation of clauses in parser, optionally delegate groups to FieldType.getSetQuery

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)

      Description

      Solr's query parser currently creates a query for each clause it sees independently. This is no longer desirable, since if we are creating a filter (non-scoring query), we can use a TermsQuery.

      PointFields SOLR-8396 also need to see a set of values together to efficiently create a set query.

      This issue:

      • adds context for the parser to know when it's parsing a query for matching only (that's only for the TermsQuery... I assume Points would always use a set query when passed multiple values)
      • defers creation of real Query objects for simple term queries in a boolean query
      • aggregates those values by field at the end to be passed to a new FieldType.getSetQuery method
      • FieldType.getSetQuery base implementation uses TermsQuery for indexed fields
      1. SOLR-9786.patch
        35 kB
        Yonik Seeley
      2. SOLR-9786.patch
        27 kB
        Yonik Seeley

        Issue Links

          Activity

          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Draft patch attached.

          I also plan on doing some benchmarking of query parsing to ensure we're not slowing anything down too much.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - Draft patch attached. I also plan on doing some benchmarking of query parsing to ensure we're not slowing anything down too much.
          Hide
          dsmiley David Smiley added a comment -

          This is no longer desirable

          That wording suggests to me there was some change to something that made the current code inefficient... but my understanding here is instead this is really just an optimization that could have actually been added a long time ago (remember TermsFilter, the Solr 4 version of TermsQuery). Right?

          I noticed the QParser.setIsFilter this patch added. This addresses an old issue I filed: SOLR-2883 What do you think of the local-params approach suggested there?

          Show
          dsmiley David Smiley added a comment - This is no longer desirable That wording suggests to me there was some change to something that made the current code inefficient... but my understanding here is instead this is really just an optimization that could have actually been added a long time ago (remember TermsFilter, the Solr 4 version of TermsQuery). Right? I noticed the QParser.setIsFilter this patch added. This addresses an old issue I filed: SOLR-2883 What do you think of the local-params approach suggested there?
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          > This is no longer desirable
          That wording suggests to me there was some change to something that made the current code inefficient... but my understanding here is instead this is really just an optimization that could have actually been added a long time ago

          I was speaking from the perspective of when the QueryParser was first created... there was no reason to not just immediately create a query for each clause. Compared to now, when we have query types that can do set queries more efficiently.

          SOLR-2883 What do you think of the local-params approach suggested there?

          If we needed a local param for it, that could set the right flag. But I'm not sure it is needed? filter queries should always ignore scores, and in a relevance query, one can always use something like filter(). Ultimately I think this should be an input to the qparser constructor... but that can probably wait for some refactoring when JSON query parsing comes along in an integrated form with QParsers.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - > This is no longer desirable That wording suggests to me there was some change to something that made the current code inefficient... but my understanding here is instead this is really just an optimization that could have actually been added a long time ago I was speaking from the perspective of when the QueryParser was first created... there was no reason to not just immediately create a query for each clause. Compared to now, when we have query types that can do set queries more efficiently. SOLR-2883 What do you think of the local-params approach suggested there? If we needed a local param for it, that could set the right flag. But I'm not sure it is needed? filter queries should always ignore scores, and in a relevance query, one can always use something like filter(). Ultimately I think this should be an input to the qparser constructor... but that can probably wait for some refactoring when JSON query parsing comes along in an integrated form with QParsers.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Patch w/ tests

          Show
          yseeley@gmail.com Yonik Seeley added a comment - Patch w/ tests
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bf9db95f218f49bac8e7971eb953a9fd9d13a2f0 in lucene-solr's branch refs/heads/master from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bf9db95 ]

          SOLR-9786: FieldType.getSetQuery, QParser.flags/isFilter, solr query parser use getSetQuery when appropriate and other optimizations

          Show
          jira-bot ASF subversion and git services added a comment - Commit bf9db95f218f49bac8e7971eb953a9fd9d13a2f0 in lucene-solr's branch refs/heads/master from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bf9db95 ] SOLR-9786 : FieldType.getSetQuery, QParser.flags/isFilter, solr query parser use getSetQuery when appropriate and other optimizations
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9f24c4b59a89bb1d42b2e93abbcfb4076777e321 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9f24c4b ]

          SOLR-9786: FieldType.getSetQuery, QParser.flags/isFilter, solr query parser use getSetQuery when appropriate and other optimizations

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9f24c4b59a89bb1d42b2e93abbcfb4076777e321 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9f24c4b ] SOLR-9786 : FieldType.getSetQuery, QParser.flags/isFilter, solr query parser use getSetQuery when appropriate and other optimizations
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          I maded the isFilter + flags stuff on the QParser to be experimental so we still have flexibility to change it later.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - I maded the isFilter + flags stuff on the QParser to be experimental so we still have flexibility to change it later.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3b4cc1bd1f691446008e2947e71ce0d3bc4e91ef in lucene-solr's branch refs/heads/master from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3b4cc1b ]

          SOLR-9786: set filter flag for more parsers

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3b4cc1bd1f691446008e2947e71ce0d3bc4e91ef in lucene-solr's branch refs/heads/master from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3b4cc1b ] SOLR-9786 : set filter flag for more parsers
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e7aa0e4ce4283cc7ff535111c03830adb80da990 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e7aa0e4 ]

          SOLR-9786: set filter flag for more parsers

          Show
          jira-bot ASF subversion and git services added a comment - Commit e7aa0e4ce4283cc7ff535111c03830adb80da990 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e7aa0e4 ] SOLR-9786 : set filter flag for more parsers
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7e6e68900b8ef4199f6cee6b262544ccab0c9294 in lucene-solr's branch refs/heads/master from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7e6e689 ]

          SOLR-9786: set filter for join query parser

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7e6e68900b8ef4199f6cee6b262544ccab0c9294 in lucene-solr's branch refs/heads/master from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7e6e689 ] SOLR-9786 : set filter for join query parser
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9a980833dff47fdba003b9d5bcf4a2836f4f0c8b in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a98083 ]

          SOLR-9786: set filter for join query parser

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9a980833dff47fdba003b9d5bcf4a2836f4f0c8b in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a98083 ] SOLR-9786 : set filter for join query parser
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7d7e5d2246d69843f259b9815332a24dc621d9e7 in lucene-solr's branch refs/heads/master from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7d7e5d2 ]

          SOLR-9786: additional test related to TermInSetQuery now requiring all terms in same field

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7d7e5d2246d69843f259b9815332a24dc621d9e7 in lucene-solr's branch refs/heads/master from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7d7e5d2 ] SOLR-9786 : additional test related to TermInSetQuery now requiring all terms in same field
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3c2e8f996f81af1b7611e77ad040e33fddf80ab0 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3c2e8f9 ]

          SOLR-9786: additional test related to TermInSetQuery now requiring all terms in same field

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3c2e8f996f81af1b7611e77ad040e33fddf80ab0 in lucene-solr's branch refs/heads/branch_6x from Yonik Seeley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3c2e8f9 ] SOLR-9786 : additional test related to TermInSetQuery now requiring all terms in same field
          Hide
          hossman Hoss Man added a comment -

          Bulk editing 27 jiras matching the following JQL query to include the "numeric-tries-to-points label...

          (issue in linkedIssues(SOLR-10807) OR issue in linkedIssues(SOLR-8396) OR issue in linkedIssues(SOLR-9995)) AND project = SOLR AND status != CLOSED
          

          these 27 jiras can be identified by searching for the comment string "20170720hossbulkeditlabel"

          email notification is being suppressed.

          Show
          hossman Hoss Man added a comment - Bulk editing 27 jiras matching the following JQL query to include the "numeric-tries-to-points label... (issue in linkedIssues(SOLR-10807) OR issue in linkedIssues(SOLR-8396) OR issue in linkedIssues(SOLR-9995)) AND project = SOLR AND status != CLOSED these 27 jiras can be identified by searching for the comment string "20170720hossbulkeditlabel" email notification is being suppressed.

            People

            • Assignee:
              yseeley@gmail.com Yonik Seeley
              Reporter:
              yseeley@gmail.com Yonik Seeley
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development