Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7829

Pivot Facet Bug: facet.missing=true + facet.sort=index facet.pivot.mincount > ? == incorrect "missing" count

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.3, 6.0
    • None
    • None

    Description

      Following up on SOLR-7804 lead to this error which i'm splitting off into it's own issue. the nuthsell is that if you combine facet.missing, facet.pivot.mincount, and facet.sort=index you might get incorrect counts (or no counts at all) for the missing value of a pivot.

      fairly easy to reproduce the most extreme aspect of the problem (not getting a missing count back even though it's above the minumum)...

      bin/solr -e cloud -noprompt
      bin/post -c gettingstarted example/exampledocs/*.xml
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17},
              {
                "field":"inStock",
                "value":false,
                "count":4},
              {
                "field":"inStock",
                "value":null,
                "count":11}]}}}
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true&facet.pivot.mincount=10
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17},
              {
                "field":"inStock",
                "value":null,
                "count":11}]}}}
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true&facet.pivot.mincount=10&facet.sort=index
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17}]}}}
      
      

      ...note that in the last example, the 'null' count is gone (even though it's above the minimum) just because we changed the facet.sort.

      Attachments

        1. SOLR-7829.patch
          8 kB
          Chris M. Hostetter

        Issue Links

          Activity

            Huh.

            Well apparently the pivot refinement code has incorrectly never bothered to refining the "missing" count ... weird.

            Because the missing count isn't affected by the facet.limit, this only affects indexes & queries where facet.sort=index and at least one shard has a missing count that falls below the "per shard" mincount (facet.mincount / numShards).

            The attached patch has...

            • DistributedFacetPivotSmallTest
              • trivially little change that i made the harden that test – doesn't catch this error because the number of docs is too small
            • DistributedFacetPivotLargeTest
              • non-randomized test that demonstrates the problem
            • PivotFacetField
              • fix for the bug to do proper refinement of the missing count

            I'll let my laptop hammer on this over the weekend and plan to commit monday.

            hossman Chris M. Hostetter added a comment - Huh. Well apparently the pivot refinement code has incorrectly never bothered to refining the "missing" count ... weird. Because the missing count isn't affected by the facet.limit, this only affects indexes & queries where facet.sort=index and at least one shard has a missing count that falls below the "per shard" mincount (facet.mincount / numShards). The attached patch has... DistributedFacetPivotSmallTest trivially little change that i made the harden that test – doesn't catch this error because the number of docs is too small DistributedFacetPivotLargeTest non-randomized test that demonstrates the problem PivotFacetField fix for the bug to do proper refinement of the missing count I'll let my laptop hammer on this over the weekend and plan to commit monday.

            Commit 1692983 from hossman@apache.org in branch 'dev/trunk'
            [ https://svn.apache.org/r1692983 ]

            SOLR-7829: Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1

            jira-bot ASF subversion and git services added a comment - Commit 1692983 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1692983 ] SOLR-7829 : Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1

            Commit 1692987 from hossman@apache.org in branch 'dev/branches/branch_5x'
            [ https://svn.apache.org/r1692987 ]

            SOLR-7829: Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1 (merge r1692983)

            jira-bot ASF subversion and git services added a comment - Commit 1692987 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692987 ] SOLR-7829 : Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1 (merge r1692983)

            Bulk close for 5.3.0 release

            shalin Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

            People

              hossman Chris M. Hostetter
              hossman Chris M. Hostetter
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: