Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5260

Faceting on a field in SolrCloud with docValues=true and indexed=false fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.4
    • Fix Version/s: 6.4, 7.0
    • Component/s: search, SolrCloud
    • Labels:
      None

      Description

      I have a problem doing facet search on a doc value field in a multi shard collection.

      My Solr schema specifies fieldA as a docvalue type and I have created a two shard collection using Solr 4.4.0 (and the unreleased 4.5 branch).
      When I do a facet search on fieldA with a "large" facet.limit then the query fails with the below exception
      A "large" facet.limit seems to be when (10 + (facet.limit * 1,5)) * number of shards > rows matching my query

      The exception does not occur when I run with a single shard collection.
      It can easily be reproduced by indexing a single row and querying it, as the default facet.limit is 100.

      The facet query received by Solr looks as follows:

          576793 [qtp170860084-18] INFO  org.apache.solr.core.SolrCore  ¦ [trym_shard2_replica1] webapp=/solr path=/select 
           params={facet=true&start=0&q=*:*&distrib=true&collection=trym&facet.field=fieldA&wt=javabin&version=2&rows=0} 
           status=500 QTime=20
      

      One of the "internal query" send by Solr to its shard looks like

          576783 [qtp170860084-19] INFO  org.apache.solr.core.SolrCore  ¦ [trym_shard1_replica1] webapp=/solr path=/select 
           params={facet=true&distrib=false&collection=trym&wt=javabin&version=2&rows=0&NOW=1379855011787    
             &shard.url=192.168.56.1:8501/solr/trym_shard1_replica1/&df=text&fl=id,score&f.fieldA.facet.limit=160
             &start=0&q=*:*&facet.field=fieldA&isShard=true&fsv=true} 
           hits=1 status=500 QTime=2
      

      The exception thrown by Solr is as follows

      576784 [qtp170860084-17] ERROR org.apache.solr.servlet.SolrDispatchFilter  ¦ null:java.lang.IllegalStateException: 
           Cannot use facet.mincount=0 on a field which is not indexed
              at org.apache.solr.request.NumericFacets.getCounts(NumericFacets.java:257)
              at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:423)
              at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:530)
              at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:259)
              at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:78)
              at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
              at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
              at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
              at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
              at org.eclipse.jetty.server.Server.handle(Server.java:368)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
              at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
              at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
              at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
              at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
              at java.lang.Thread.run(Thread.java:724)
      

      My schema.xml contains the following lines (among others )

          <dynamicField name="*A" type="dlong" indexed="false" stored="true" docValues="true" required="true"/>
          ...
          <fieldType name="dlong" class="solr.TrieLongField" docValuesFormat="Disk" precisionStep="0" positionIncrementGap="0"/>
      
      1. SOLR-5260.patch
        22 kB
        Erick Erickson
      2. SOLR-5260-testfix.patch
        4 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          trym Trym Møller added a comment -

          This only seems to be the case when faceting on a numeric field. Why SimpleFacets.java wants it in that way is not clear:

           if (ft.getNumericType() != null && sf.hasDocValues()) {
                // only fcs is able to leverage the numeric field caches
                method = FacetMethod.FCS;
              }
          
          Show
          trym Trym Møller added a comment - This only seems to be the case when faceting on a numeric field. Why SimpleFacets.java wants it in that way is not clear: if (ft.getNumericType() != null && sf.hasDocValues()) { // only fcs is able to leverage the numeric field caches method = FacetMethod.FCS; }
          Hide
          erickerickson Erick Erickson added a comment - - edited

          This is on trunk, fresh pull.

          I was looking at this today and it's a problem indeed. Since we're advocating using docValues for faceting failing sometimes and succeeding others is disconcerting.

          I have a field indexed="false" docValues="true". Sometimes it works and sometimes it doesn't, it depends as Trym says, on how many docs are in the result set and the number of shards. The error is reported as Trym indicated, even on a current trunk.

          Caused by: java.lang.IllegalStateException: Cannot use facet.mincount=0 on field eoe which is not indexed
          at org.apache.solr.request.NumericFacets.getCounts(NumericFacets.java:256)
          at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:465)

          json facets work with mincount>0. If mincount=0 it fails with an error message (on the client and the server) something like:
          "Numeric fields do not support facet mincount=0; try indexing as terms".

          Anyway, I'm surely not going to get to this in the near future, so un-assigning it to myself. Not something for 6.3 as it's been around for a long time.

          I guess there are two work-arounds at present:
          1> use json facets
          or
          2> set index=true

          Show
          erickerickson Erick Erickson added a comment - - edited This is on trunk, fresh pull. I was looking at this today and it's a problem indeed. Since we're advocating using docValues for faceting failing sometimes and succeeding others is disconcerting. I have a field indexed="false" docValues="true". Sometimes it works and sometimes it doesn't, it depends as Trym says, on how many docs are in the result set and the number of shards. The error is reported as Trym indicated, even on a current trunk. Caused by: java.lang.IllegalStateException: Cannot use facet.mincount=0 on field eoe which is not indexed at org.apache.solr.request.NumericFacets.getCounts(NumericFacets.java:256) at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:465) json facets work with mincount>0. If mincount=0 it fails with an error message (on the client and the server) something like: "Numeric fields do not support facet mincount=0; try indexing as terms". Anyway, I'm surely not going to get to this in the near future, so un-assigning it to myself. Not something for 6.3 as it's been around for a long time. I guess there are two work-arounds at present: 1> use json facets or 2> set index=true
          Hide
          erickerickson Erick Erickson added a comment -

          Assigning to myself to not lose track of it. I certainly won't get to it promptly so anyone please feel free.

          Show
          erickerickson Erick Erickson added a comment - Assigning to myself to not lose track of it. I certainly won't get to it promptly so anyone please feel free.
          Hide
          erickerickson Erick Erickson added a comment -

          Patch. The actual code change is one line, the rest is testing.

          Tried a new approach to getting custom schemas. Rather than create yet another schema file I used the managed schema api to create what I needed in @BeforeClass.

          Show
          erickerickson Erick Erickson added a comment - Patch. The actual code change is one line, the rest is testing. Tried a new approach to getting custom schemas. Rather than create yet another schema file I used the managed schema api to create what I needed in @BeforeClass.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ca6626604b968eeb11630b0fdf64178a71f3999b in lucene-solr's branch refs/heads/master from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ca66266 ]

          SOLR-5260: Faceting on a field in SolrCloud with docValues=true and indexed=false fails

          Show
          jira-bot ASF subversion and git services added a comment - Commit ca6626604b968eeb11630b0fdf64178a71f3999b in lucene-solr's branch refs/heads/master from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ca66266 ] SOLR-5260 : Faceting on a field in SolrCloud with docValues=true and indexed=false fails
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit af904f8dd2a5e3001fbd764c474c745b7b34147c in lucene-solr's branch refs/heads/branch_6x from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=af904f8 ]

          SOLR-5260: Faceting on a field in SolrCloud with docValues=true and indexed=false fails
          (cherry picked from commit ca66266)

          Show
          jira-bot ASF subversion and git services added a comment - Commit af904f8dd2a5e3001fbd764c474c745b7b34147c in lucene-solr's branch refs/heads/branch_6x from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=af904f8 ] SOLR-5260 : Faceting on a field in SolrCloud with docValues=true and indexed=false fails (cherry picked from commit ca66266)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e4733f696eba31e6903da140ae411bccff996b6e in lucene-solr's branch refs/heads/master from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e4733f6 ]

          SOLR-5260: Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix

          Show
          jira-bot ASF subversion and git services added a comment - Commit e4733f696eba31e6903da140ae411bccff996b6e in lucene-solr's branch refs/heads/master from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e4733f6 ] SOLR-5260 : Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 500c2befe98cce2a2619d1022054d7e75c68716d in lucene-solr's branch refs/heads/master from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=500c2be ]

          SOLR-5260: Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix

          Show
          jira-bot ASF subversion and git services added a comment - Commit 500c2befe98cce2a2619d1022054d7e75c68716d in lucene-solr's branch refs/heads/master from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=500c2be ] SOLR-5260 : Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 523b6b1b9f199b854642714e7b3d23e9457115d7 in lucene-solr's branch refs/heads/branch_6x from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=523b6b1 ]

          SOLR-5260: Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix
          (cherry picked from commit e4733f6)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 523b6b1b9f199b854642714e7b3d23e9457115d7 in lucene-solr's branch refs/heads/branch_6x from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=523b6b1 ] SOLR-5260 : Faceting on a field in SolrCloud with docValues=true and indexed=false fails, test fix (cherry picked from commit e4733f6)
          Hide
          dsmiley David Smiley added a comment -

          Nice testing technique to not create yet another test schema, Erick. I think I may approach this similarly in future tests. I wonder how succinct it could be made if there's just a few fields needed. Clearly this tests requires a lot and I see why there is so much code devoted to defining the schema for this test.

          One little quibble. Instead of Stream.of(...).collect(Collectors.toList(), I think it's much more common (and thus clear) to simply use Arrays.asList(...). Also, using SimpleDateFormat for the ISO-8601 format we use in Solr is much more wordy way of doing it than Java 8 Instant... in particular the one line where you reference the format would then look like this:
          return Instant.ofEpochMilli(985_847_645 + (long) base + counter).toString();

          Show
          dsmiley David Smiley added a comment - Nice testing technique to not create yet another test schema, Erick. I think I may approach this similarly in future tests. I wonder how succinct it could be made if there's just a few fields needed. Clearly this tests requires a lot and I see why there is so much code devoted to defining the schema for this test. One little quibble. Instead of Stream.of(...).collect(Collectors.toList() , I think it's much more common (and thus clear) to simply use Arrays.asList(...) . Also, using SimpleDateFormat for the ISO-8601 format we use in Solr is much more wordy way of doing it than Java 8 Instant... in particular the one line where you reference the format would then look like this: return Instant.ofEpochMilli(985_847_645 + (long) base + counter).toString();
          Hide
          erickerickson Erick Erickson added a comment - - edited

          David:

          Yeah, still not up to the Java8 Instant stuff, I'll put that in my bag of tricks.

          About defining fields on the fly. One of my questions is whether this could be abstracted into a more consumable form and, maybe, put in SolrCloudTestCase? Or even static methods in a utility class? There was a lot of code there as you say just to get my fields defined. Some of the bloat is because I needed a way to predict what the sort order would be but still wanted some randomness, and partly I thought it a PoC....

          And did I really check in code with //nocommit in it? Yes I did. Git and I are not friends, a merge problem 'cause trunk is OK.. Fixing shortly.

          Show
          erickerickson Erick Erickson added a comment - - edited David: Yeah, still not up to the Java8 Instant stuff, I'll put that in my bag of tricks. About defining fields on the fly. One of my questions is whether this could be abstracted into a more consumable form and, maybe, put in SolrCloudTestCase? Or even static methods in a utility class? There was a lot of code there as you say just to get my fields defined. Some of the bloat is because I needed a way to predict what the sort order would be but still wanted some randomness, and partly I thought it a PoC.... And did I really check in code with //nocommit in it? Yes I did. Git and I are not friends, a merge problem 'cause trunk is OK.. Fixing shortly.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 736c9ff8e33019ba5f29f464629ad501f915b383 in lucene-solr's branch refs/heads/branch_6x from Erick Erickson
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=736c9ff ]

          SOLR-5260 merge error

          Show
          jira-bot ASF subversion and git services added a comment - Commit 736c9ff8e33019ba5f29f464629ad501f915b383 in lucene-solr's branch refs/heads/branch_6x from Erick Erickson [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=736c9ff ] SOLR-5260 merge error
          Hide
          erickerickson Erick Erickson added a comment -

          Attached a testfix patch. Does NOT have the //nocommit I managed to put in 6x.

          Sorry for the noise.

          Show
          erickerickson Erick Erickson added a comment - Attached a testfix patch. Does NOT have the //nocommit I managed to put in 6x. Sorry for the noise.
          Hide
          tomasflobbe Tomás Fernández Löbbe added a comment -

          The issue here is that a numeric field indexed=false and docValues=true doesn't really work with facet.mincount=0 (distributed request or not). Values with count=0 are not returned.

          Show
          tomasflobbe Tomás Fernández Löbbe added a comment - The issue here is that a numeric field indexed=false and docValues=true doesn't really work with facet.mincount=0 (distributed request or not). Values with count=0 are not returned.

            People

            • Assignee:
              erickerickson Erick Erickson
              Reporter:
              trym Trym Møller
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development