Solr
  1. Solr
  2. SOLR-3177

Excluding tagged filter in StatsComponent

    Details

      Description

      It would be useful to exclude the effects of some "fq" params from the set of documents used to compute stats – similar to
      how you can exclude tagged filters when generating facet counts...

      https://wiki.apache.org/solr/SimpleFacetParameters#Tagging_and_excluding_Filters

      So that it's possible to do something like this...

      http://localhost:8983/solr/select?fq=

      {!tag=priceFilter}

      price:[1 TO 20]q=:&stats=true&stats.field=

      {!ex=priceFilter}

      price

      If you want to create a price slider this is very useful because then you can filter the price ([1 TO 20) and nevertheless get the lower and upper bound of the unfiltered price (min=0, max=100):

      |<-[-------]-------------->|
      $0 $1     $20            $100
      
      1. SOLR-3177.patch
        9 kB
        Vitaliy Zhovtyuk
      2. SOLR-3177.patch
        10 kB
        Vitaliy Zhovtyuk
      3. SOLR-3177.patch
        10 kB
        Nikolai Luthman

        Activity

        Hide
        CP added a comment -

        This feature is also necessary while using multi-select range facets with facet.range to get min and max of a field to set facet.range.start and facet.range.end.

        Show
        CP added a comment - This feature is also necessary while using multi-select range facets with facet.range to get min and max of a field to set facet.range.start and facet.range.end.
        Hide
        Roel Arents added a comment - - edited

        This would indeed be useful. The only alternative I see now besides having a static min and max, is firing two queries; one with and one without the fq for the price slider.

        It seems to me that a large part of the code from solr.request.SimpleFacets beyond this line can be re-used in solr.handler.component.StatsComponent. (Too bad I'm not capable enough...)

        // figure out if we need a new base DocSet
            String excludeStr = localParams.get(CommonParams.EXCLUDE);
            if (excludeStr == null) return;
        
        Show
        Roel Arents added a comment - - edited This would indeed be useful. The only alternative I see now besides having a static min and max, is firing two queries; one with and one without the fq for the price slider. It seems to me that a large part of the code from solr.request.SimpleFacets beyond this line can be re-used in solr.handler.component.StatsComponent. (Too bad I'm not capable enough...) // figure out if we need a new base DocSet String excludeStr = localParams.get(CommonParams.EXCLUDE); if (excludeStr == null ) return ;
        Hide
        Saroj Kumar added a comment -

        Yes, this feature would be really nice for use cases where you want to get stats excluding filters

        Show
        Saroj Kumar added a comment - Yes, this feature would be really nice for use cases where you want to get stats excluding filters
        Hide
        Nikolai Luthman added a comment -

        I've made a patch for this, based on the code from the FacetComponent. The patch is for 3.6.1.

        Might need some cleanup to get it into the latest version.

        Apply by changing to solr/core/src/java/org/apache/solr/handler/component/ and run:
        patch < statsfilterexclude.patch

        Show
        Nikolai Luthman added a comment - I've made a patch for this, based on the code from the FacetComponent. The patch is for 3.6.1. Might need some cleanup to get it into the latest version. Apply by changing to solr/core/src/java/org/apache/solr/handler/component/ and run: patch < statsfilterexclude.patch
        Hide
        Jan Høydahl added a comment -

        Thanks for the patch, Nikolai.

        You'll have a greater chance of having the patch committed if you bring it even closer to finalization. The patch should ideally be against TRUNK, alternatively against branch_4x. Also, please review this page http://wiki.apache.org/solr/HowToContribute#Generating_a_patch and try to follow the guidelines as closely as possible.

        Most important I think is that the patch only includes needed changes so that it is really easy to review what has changed. If you can write JUnit tests that's perfect, if not that's ok too

        Show
        Jan Høydahl added a comment - Thanks for the patch, Nikolai. You'll have a greater chance of having the patch committed if you bring it even closer to finalization. The patch should ideally be against TRUNK, alternatively against branch_4x. Also, please review this page http://wiki.apache.org/solr/HowToContribute#Generating_a_patch and try to follow the guidelines as closely as possible. Most important I think is that the patch only includes needed changes so that it is really easy to review what has changed. If you can write JUnit tests that's perfect, if not that's ok too
        Hide
        Nikolai Luthman added a comment -

        Thanks for the comment Jan. I've updated the patch now to work against trunk, and also added a testcase that tests the !ex and !key functionality.

        Show
        Nikolai Luthman added a comment - Thanks for the comment Jan. I've updated the patch now to work against trunk, and also added a testcase that tests the !ex and !key functionality.
        Hide
        Alexander Buhr added a comment -

        is this going to be released at some point?

        Show
        Alexander Buhr added a comment - is this going to be released at some point?
        Hide
        Mircea Pop added a comment -

        In order to solve this silder problem, one can also use the group mechanism together with stats. Check this answer http://stackoverflow.com/questions/16518408/solr-query-to-get-the-overall-stats-not-just-the-filtered-ones/16538477#16538477

        Show
        Mircea Pop added a comment - In order to solve this silder problem, one can also use the group mechanism together with stats. Check this answer http://stackoverflow.com/questions/16518408/solr-query-to-get-the-overall-stats-not-just-the-filtered-ones/16538477#16538477
        Hide
        Vitaly Lazutin added a comment -

        Grouping is not the best solution.
        If you will remove fq with this field, then your facet results will be wrong. Facets on other fields will return more results, then actually need.

        Show
        Vitaly Lazutin added a comment - Grouping is not the best solution. If you will remove fq with this field, then your facet results will be wrong. Facets on other fields will return more results, then actually need.
        Hide
        Uli Kasper added a comment -

        is there any solution implemented on solr 4.4?
        I have the same issue as OP. ... want to use a price range slider and need priceFilter excluded from stats.field.
        What about that patch? someone tried to apply it on solr?

        Show
        Uli Kasper added a comment - is there any solution implemented on solr 4.4? I have the same issue as OP. ... want to use a price range slider and need priceFilter excluded from stats.field. What about that patch? someone tried to apply it on solr?
        Hide
        Dmitriy Garanzha added a comment -

        I have tried apply to 4.6 - failed(

        Show
        Dmitriy Garanzha added a comment - I have tried apply to 4.6 - failed(
        Hide
        Vitaliy Zhovtyuk added a comment -

        Updated to latest trunk. Fixed StatsComponent to pass tests.

        Show
        Vitaliy Zhovtyuk added a comment - Updated to latest trunk. Fixed StatsComponent to pass tests.
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Nikolai and Vitaliy.

        There is a bunch of code related to grouping copied from SimpleFacets which probably does not belong:

        if (rb.grouping() && rb.getGroupingSpec().isTruncateGroups()) {
                Grouping grouping = new Grouping(searcher, null, rb.getQueryCommand(), false, 0, false);
                grouping.setGroupSort(rb.getGroupingSpec().getSortWithinGroup());
                if (rb.getGroupingSpec().getFields().length > 0) {
                  grouping.addFieldCommand(rb.getGroupingSpec().getFields()[0], req);
                } else if (rb.getGroupingSpec().getFunctions().length > 0) {
                  grouping.addFunctionCommand(rb.getGroupingSpec().getFunctions()[0], req);
                } else {
                  this.base = base;
                  return;
                }
                AbstractAllGroupHeadsCollector allGroupHeadsCollector = grouping.getCommands().get(0).createAllGroupCollector();
                searcher.search(new MatchAllDocsQuery(), base.getTopFilter(), allGroupHeadsCollector);
                int maxDoc = searcher.maxDoc();
                FixedBitSet fixedBitSet = allGroupHeadsCollector.retrieveGroupHeads(maxDoc);
                long[] bits = fixedBitSet.getBits();
                this.base = new BitDocSet(new FixedBitSet(bits, bits.length));
              }
        

        If this is indeed what it takes to have StatsComponent play well with grouping then at a minimum we must have tests for it. In any case, adding support for grouping in stats computations warrants a new issue.

        Show
        Shalin Shekhar Mangar added a comment - Thanks Nikolai and Vitaliy. There is a bunch of code related to grouping copied from SimpleFacets which probably does not belong: if (rb.grouping() && rb.getGroupingSpec().isTruncateGroups()) { Grouping grouping = new Grouping(searcher, null , rb.getQueryCommand(), false , 0, false ); grouping.setGroupSort(rb.getGroupingSpec().getSortWithinGroup()); if (rb.getGroupingSpec().getFields().length > 0) { grouping.addFieldCommand(rb.getGroupingSpec().getFields()[0], req); } else if (rb.getGroupingSpec().getFunctions().length > 0) { grouping.addFunctionCommand(rb.getGroupingSpec().getFunctions()[0], req); } else { this .base = base; return ; } AbstractAllGroupHeadsCollector allGroupHeadsCollector = grouping.getCommands().get(0).createAllGroupCollector(); searcher.search( new MatchAllDocsQuery(), base.getTopFilter(), allGroupHeadsCollector); int maxDoc = searcher.maxDoc(); FixedBitSet fixedBitSet = allGroupHeadsCollector.retrieveGroupHeads(maxDoc); long [] bits = fixedBitSet.getBits(); this .base = new BitDocSet( new FixedBitSet(bits, bits.length)); } If this is indeed what it takes to have StatsComponent play well with grouping then at a minimum we must have tests for it. In any case, adding support for grouping in stats computations warrants a new issue.
        Hide
        Vitaliy Zhovtyuk added a comment -

        Removed grouping code from org.apache.solr.handler.component.SimpleStats#parseParams.

        Grouping functionality in stats need ticket to address it separately.

        Show
        Vitaliy Zhovtyuk added a comment - Removed grouping code from org.apache.solr.handler.component.SimpleStats#parseParams. Grouping functionality in stats need ticket to address it separately.
        Hide
        ASF subversion and git services added a comment -

        Commit 1577976 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1577976 ]

        SOLR-3177: Enable tagging and excluding filters in StatsComponent via the localParams syntax

        Show
        ASF subversion and git services added a comment - Commit 1577976 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1577976 ] SOLR-3177 : Enable tagging and excluding filters in StatsComponent via the localParams syntax
        Hide
        ASF subversion and git services added a comment -

        Commit 1577977 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1577977 ]

        SOLR-3177: Enable tagging and excluding filters in StatsComponent via the localParams syntax

        Show
        ASF subversion and git services added a comment - Commit 1577977 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1577977 ] SOLR-3177 : Enable tagging and excluding filters in StatsComponent via the localParams syntax
        Hide
        Shalin Shekhar Mangar added a comment -

        This will be released with Solr 4.8

        Thank you all for the comments and upvotes and sorry that this took so much time. Thanks Nikolai and Vitaliy for the patches!

        Show
        Shalin Shekhar Mangar added a comment - This will be released with Solr 4.8 Thank you all for the comments and upvotes and sorry that this took so much time. Thanks Nikolai and Vitaliy for the patches!
        Hide
        Uwe Schindler added a comment -

        Close issue after release of 4.8.0

        Show
        Uwe Schindler added a comment - Close issue after release of 4.8.0

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Mathias H.
          • Votes:
            20 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development