Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9979

Macro expansion should not be done in shard requests

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Macro expansion is done when parsing the query for the first time, it should not be re-done in shard requests.

      1. SOLR-9979.patch
        7 kB
        Tomás Fernández Löbbe

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 68d246df003278ba0c35ae5f43872340b676a02f in lucene-solr's branch refs/heads/master from Tomas Fernandez Lobbe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=68d246d ]

        SOLR-9979: Macro expansion should not be done in shard requests

        Show
        jira-bot ASF subversion and git services added a comment - Commit 68d246df003278ba0c35ae5f43872340b676a02f in lucene-solr's branch refs/heads/master from Tomas Fernandez Lobbe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=68d246d ] SOLR-9979 : Macro expansion should not be done in shard requests
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 0e728461e9a46d0e17d0b7a262648a6732b31cf0 in lucene-solr's branch refs/heads/branch_6x from Tomas Fernandez Lobbe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0e72846 ]

        SOLR-9979: Macro expansion should not be done in shard requests

        Show
        jira-bot ASF subversion and git services added a comment - Commit 0e728461e9a46d0e17d0b7a262648a6732b31cf0 in lucene-solr's branch refs/heads/branch_6x from Tomas Fernandez Lobbe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0e72846 ] SOLR-9979 : Macro expansion should not be done in shard requests
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c58eac13378f618532190348574d96a72ef413e7 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c58eac1 ]

        SOLR-9977, SOLR-9979: move CHANGES entries to correct section (Bug)

        Aparently these were shifted during merge/cherry-picks

        Show
        jira-bot ASF subversion and git services added a comment - Commit c58eac13378f618532190348574d96a72ef413e7 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c58eac1 ] SOLR-9977 , SOLR-9979 : move CHANGES entries to correct section (Bug) Aparently these were shifted during merge/cherry-picks
        Hide
        alessandro.benedetti Alessandro Benedetti added a comment -

        Hi Tomás Fernández Löbbe,
        what happens now in this scenario :

        1) Manual Distributed Search with not uniform data model across shards.

        q=$

        {pop_query}

        &pop_query=$

        {pop_field}

        :[ $

        {low}

        TO $

        {high}

        ] AND inStock:true
        &low=50
        &high=100

        Each shard has a different popularity field, each shard defines this field as a parameter substitution :
        Shard1 Request Handler
        <str name="pop_field">popularity</str>

        Shard 2 Request Handler
        <str name="pop_field">clicks</str>

        2) SolrCloud inter collection search
        Same scenario, but each collection may define a different mapping.

        Thanks,

        Regards

        Show
        alessandro.benedetti Alessandro Benedetti added a comment - Hi Tomás Fernández Löbbe , what happens now in this scenario : 1) Manual Distributed Search with not uniform data model across shards. q=$ {pop_query} &pop_query=$ {pop_field} :[ $ {low} TO $ {high} ] AND inStock:true &low=50 &high=100 Each shard has a different popularity field, each shard defines this field as a parameter substitution : Shard1 Request Handler <str name="pop_field">popularity</str> Shard 2 Request Handler <str name="pop_field">clicks</str> 2) SolrCloud inter collection search Same scenario, but each collection may define a different mapping. Thanks, Regards
        Hide
        alessandro.benedetti Alessandro Benedetti added a comment -

        Hi Tomás Fernández Löbbe,
        what happens now in this scenario :

        1) Manual Distributed Search with not uniform data model across shards.

        q=$

        {pop_query}

        &pop_query=$

        {pop_field}

        :[ $

        {low}

        TO $

        {high}

        ] AND inStock:true
        &low=50
        &high=100

        Each shard has a different popularity field, each shard defines this field as a parameter substitution :
        Shard1 Request Handler
        <str name="pop_field">popularity</str>

        Shard 2 Request Handler
        <str name="pop_field">clicks</str>

        2) SolrCloud inter collection search
        Same scenario, but each collection may define a different mapping.

        Thanks,

        Regards

        Show
        alessandro.benedetti Alessandro Benedetti added a comment - Hi Tomás Fernández Löbbe , what happens now in this scenario : 1) Manual Distributed Search with not uniform data model across shards. q=$ {pop_query} &pop_query=$ {pop_field} :[ $ {low} TO $ {high} ] AND inStock:true &low=50 &high=100 Each shard has a different popularity field, each shard defines this field as a parameter substitution : Shard1 Request Handler <str name="pop_field">popularity</str> Shard 2 Request Handler <str name="pop_field">clicks</str> 2) SolrCloud inter collection search Same scenario, but each collection may define a different mapping. Thanks, Regards
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        Hi Alessandro Benedetti,
        Those use cases are not supported after this change, some of it could be achieved by using parameter dereferencing with Local Params. I would argue that in general searching across collections with different configs is not supported (as there are no tests!), but feel free to open a new Jira to improve this change.

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - Hi Alessandro Benedetti , Those use cases are not supported after this change, some of it could be achieved by using parameter dereferencing with Local Params. I would argue that in general searching across collections with different configs is not supported (as there are no tests!), but feel free to open a new Jira to improve this change.
        Hide
        alessandro.benedetti Alessandro Benedetti added a comment -

        Uhm, so this patch breaks that feature, but you say that that was not a feature at all but a side effect ?
        I am still a little bit confuse, what are the benefits of this patch ?
        In my opinion the feature I mentioned was quite useful and it has been supported for a while

        Show
        alessandro.benedetti Alessandro Benedetti added a comment - Uhm, so this patch breaks that feature, but you say that that was not a feature at all but a side effect ? I am still a little bit confuse, what are the benefits of this patch ? In my opinion the feature I mentioned was quite useful and it has been supported for a while
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        you say that that was not a feature at all but a side effect ?

        Not sure if that was intentional or not. Yonik Seeley, were you thinking in this use case?

        I am still a little bit confuse, what are the benefits of this patch ?

        The issue I saw is that internal requests can add parameter that are not intended for macro expansion (for example in facet refinement), and treating them as macros can cause exceptions or incorrect results (see the test case I added).

        In my opinion the feature I mentioned was quite useful and it has been supported for a while

        We can't reopen this Jira, since it was already released, but feel free to create a new one to support your use case

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - you say that that was not a feature at all but a side effect ? Not sure if that was intentional or not. Yonik Seeley , were you thinking in this use case? I am still a little bit confuse, what are the benefits of this patch ? The issue I saw is that internal requests can add parameter that are not intended for macro expansion (for example in facet refinement), and treating them as macros can cause exceptions or incorrect results (see the test case I added). In my opinion the feature I mentioned was quite useful and it has been supported for a while We can't reopen this Jira, since it was already released, but feel free to create a new one to support your use case

          People

          • Assignee:
            tomasflobbe Tomás Fernández Löbbe
            Reporter:
            tomasflobbe Tomás Fernández Löbbe
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development