Solr
  1. Solr
  2. SOLR-7829

Pivot Facet Bug: facet.missing=true + facet.sort=index facet.pivot.mincount > ? == incorrect "missing" count

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Following up on SOLR-7804 lead to this error which i'm splitting off into it's own issue. the nuthsell is that if you combine facet.missing, facet.pivot.mincount, and facet.sort=index you might get incorrect counts (or no counts at all) for the missing value of a pivot.

      fairly easy to reproduce the most extreme aspect of the problem (not getting a missing count back even though it's above the minumum)...

      bin/solr -e cloud -noprompt
      bin/post -c gettingstarted example/exampledocs/*.xml
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17},
              {
                "field":"inStock",
                "value":false,
                "count":4},
              {
                "field":"inStock",
                "value":null,
                "count":11}]}}}
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true&facet.pivot.mincount=10
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17},
              {
                "field":"inStock",
                "value":null,
                "count":11}]}}}
      
      http://localhost:8983/solr/gettingstarted/query?rows=0&q=*:*&facet=true&facet.pivot=inStock&facet.missing=true&facet.pivot.mincount=10&facet.sort=index
      ...
          "facet_pivot":{
            "inStock":[{
                "field":"inStock",
                "value":true,
                "count":17}]}}}
      
      

      ...note that in the last example, the 'null' count is gone (even though it's above the minimum) just because we changed the facet.sort.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          Huh.

          Well apparently the pivot refinement code has incorrectly never bothered to refining the "missing" count ... weird.

          Because the missing count isn't affected by the facet.limit, this only affects indexes & queries where facet.sort=index and at least one shard has a missing count that falls below the "per shard" mincount (facet.mincount / numShards).

          The attached patch has...

          • DistributedFacetPivotSmallTest
            • trivially little change that i made the harden that test – doesn't catch this error because the number of docs is too small
          • DistributedFacetPivotLargeTest
            • non-randomized test that demonstrates the problem
          • PivotFacetField
            • fix for the bug to do proper refinement of the missing count

          I'll let my laptop hammer on this over the weekend and plan to commit monday.

          Show
          Hoss Man added a comment - Huh. Well apparently the pivot refinement code has incorrectly never bothered to refining the "missing" count ... weird. Because the missing count isn't affected by the facet.limit, this only affects indexes & queries where facet.sort=index and at least one shard has a missing count that falls below the "per shard" mincount (facet.mincount / numShards). The attached patch has... DistributedFacetPivotSmallTest trivially little change that i made the harden that test – doesn't catch this error because the number of docs is too small DistributedFacetPivotLargeTest non-randomized test that demonstrates the problem PivotFacetField fix for the bug to do proper refinement of the missing count I'll let my laptop hammer on this over the weekend and plan to commit monday.
          Hide
          ASF subversion and git services added a comment -

          Commit 1692983 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1692983 ]

          SOLR-7829: Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1

          Show
          ASF subversion and git services added a comment - Commit 1692983 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1692983 ] SOLR-7829 : Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1
          Hide
          ASF subversion and git services added a comment -

          Commit 1692987 from hossman@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1692987 ]

          SOLR-7829: Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1 (merge r1692983)

          Show
          ASF subversion and git services added a comment - Commit 1692987 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692987 ] SOLR-7829 : Fixed a bug in distributed pivot faceting that could result in a facet.missing=true count which was lower then the correct count if facet.sort=index and facet.pivot.mincount > 1 (merge r1692983)
          Hide
          Shalin Shekhar Mangar added a comment -

          Bulk close for 5.3.0 release

          Show
          Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

            People

            • Assignee:
              Hoss Man
              Reporter:
              Hoss Man
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development