Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9490

BoolField always returning false for non-DV fields when javabin involved (via solrj, or intra node communication)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.2
    • Fix Version/s: 6.2.1, 6.3, master (7.0)
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      2 diff users posted comments in SOLR-9187 indicating that changes introduced in that issue have broken BoolFields that do not use DocValues...

      Colvin Cowie...

      Hi, I've just picked up 6.2.0. It seems that the change to toExternal() in BoolField now means that booleans without DocValues return null, which then turns into Boolean.FALSE in toObject() regardless of whether the value is true or false.

      e.g. with this schema, facet counts are correct, the returned values are wrong.

      <field name="f_EVE64" type="boolean" indexed="true" stored="true" required="false" multiValued="false"/>
      <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
      
      "response":{"numFound":2,"start":0,"maxScore":1.0,"docs":[
            {
              "id":"124",
              "f_EVE64":false,
              "_version_":1544828487600177152},
            {
              "id":"123",
              "f_EVE64":false,
              "_version_":1544828492458229760}]
        },
        "facet_counts":{
          "facet_queries":{},
          "facet_fields":{
            "f_EVE64":[
              "false",1,
              "true",1]},
      

      Could toExternal() perhaps fallback to how it originally behaved? e.g.

      if (f.binaryValue() == null) {
            return indexedToReadable(f.stringValue());
      }
      

      Pavan Shetty...

      I downloaded solr version 6.2.0 (6.2.0 764d0f19151dbff6f5fcd9fc4b2682cf934590c5 - mike - 2016-08-20 05:41:37) and installed my core.

      In my schema.xml i have an field like following :

      <field name="stock_status" type="boolean" indexed="true" stored="true" multiValued="false"/>

      Now i am accessing this field using SolrJ (6.1.0). But i am always getting false value for above field even though it contains true boolean value. This is happening for all boolean fields.

      http://localhost:8983/solr...wt=javabin&version=2 HTTP/1.1

      It is working fine in other response writer.

      If i change the solr version to 6.1.0, with same SolrJ, it starts working. So clearly this is a bug in version 6.2.0.

      1. Solr9490.java
        2 kB
        Dan Fox
      2. SOLR-9490.patch
        2 kB
        Hoss Man
      3. SOLR-9490.patch
        1 kB
        Hoss Man

        Issue Links

          Activity

          Hide
          hossman Hoss Man added a comment - - edited

          I'm having trouble trying to (trivially) reproduce this ... Colvin Cowie's comments suggest this has nothing to do with the response writer used, and that the JSON output would demonstrate the same problem, but when i try the simplest possible steps to demonstrate it seems to be working fine...

          $ git co releases/lucene-solr/6.2.0
          ...
          $ ant clean && cd solr && ant server && bin/solr -e techproducts
          ...
          POSTing file solr.xml to [base]
          ...
          $ grep inStock example/exampledocs/solr.xml 
            <field name="inStock">true</field>
          $ curl 'http://localhost:8983/solr/techproducts/query?q=id:SOLR1000&fl=inStock'
          {
            "responseHeader":{
              "status":0,
              "QTime":0,
              "params":{
                "q":"id:SOLR1000",
                "fl":"inStock"}},
            "response":{"numFound":1,"start":0,"docs":[
                {
                  "inStock":true}]
            }}
          

          what am i missing?

          Colvin: what is the version attribute on your schema? does this problem exist only when you use "old" (pre-6.2) indexes after upgrading, or can folks reproduce this problem even with completely new indexes (build with 6.2) ?

          EDIT: Note this is the definition of inStock in the 6.2 techproducts schema...

          <schema name="example" version="1.6">
          ...
             <field name="inStock" type="boolean" indexed="true" stored="true" />
          ...
              <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
          
          Show
          hossman Hoss Man added a comment - - edited I'm having trouble trying to (trivially) reproduce this ... Colvin Cowie 's comments suggest this has nothing to do with the response writer used, and that the JSON output would demonstrate the same problem, but when i try the simplest possible steps to demonstrate it seems to be working fine... $ git co releases/lucene-solr/6.2.0 ... $ ant clean && cd solr && ant server && bin/solr -e techproducts ... POSTing file solr.xml to [base] ... $ grep inStock example/exampledocs/solr.xml <field name="inStock">true</field> $ curl 'http://localhost:8983/solr/techproducts/query?q=id:SOLR1000&fl=inStock' { "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"id:SOLR1000", "fl":"inStock"}}, "response":{"numFound":1,"start":0,"docs":[ { "inStock":true}] }} what am i missing? Colvin: what is the version attribute on your schema? does this problem exist only when you use "old" (pre-6.2) indexes after upgrading, or can folks reproduce this problem even with completely new indexes (build with 6.2) ? EDIT: Note this is the definition of inStock in the 6.2 techproducts schema... <schema name= "example" version= "1.6" > ... <field name= "inStock" type= " boolean " indexed= " true " stored= " true " /> ... <fieldType name= " boolean " class= "solr.BoolField" sortMissingLast= " true " />
          Hide
          pavan_shetty Pavan Shetty added a comment -

          Hoss Man, for me it seems to be the problem with solr's response writer. if i use xml or json response writer to get the information it returns proper value, true or false for boolean field, of course through solr admin console.

          But if we have binary response writer then it always returns false(or null) for boolean field, this one i tested through solr j client which uses this as response writer.

          I changed lucene version 6.1 and 6.2 in solrconfig (or solr j version 6.1 / 6.2) and it is working fine if solr version is below 6.2. if i change solr version to 6.2 then whatever lucene version (or solr j) it returns false for boolean field. Did reindexing each time i changed version of solr and lucene and issue is reproduce able for me.

          Show
          pavan_shetty Pavan Shetty added a comment - Hoss Man , for me it seems to be the problem with solr's response writer. if i use xml or json response writer to get the information it returns proper value, true or false for boolean field, of course through solr admin console. But if we have binary response writer then it always returns false(or null) for boolean field, this one i tested through solr j client which uses this as response writer. I changed lucene version 6.1 and 6.2 in solrconfig (or solr j version 6.1 / 6.2) and it is working fine if solr version is below 6.2. if i change solr version to 6.2 then whatever lucene version (or solr j) it returns false for boolean field. Did reindexing each time i changed version of solr and lucene and issue is reproduce able for me.
          Hide
          pavan_shetty Pavan Shetty added a comment -

          Also this issue is not dependent on docValues. DocValues true or false for boolean field , this issue is reproduce able.

          Show
          pavan_shetty Pavan Shetty added a comment - Also this issue is not dependent on docValues. DocValues true or false for boolean field , this issue is reproduce able.
          Hide
          erickerickson Erick Erickson added a comment -

          Hmmm, would it be at all possible to provide a unit test illustrating this? It's really helpful so we actually do the same thing.

          Then we can incorporate that test in the final patch and be sure the problem you;'re seeing is actually fixed for now and into the future.

          Show
          erickerickson Erick Erickson added a comment - Hmmm, would it be at all possible to provide a unit test illustrating this? It's really helpful so we actually do the same thing. Then we can incorporate that test in the final patch and be sure the problem you;'re seeing is actually fixed for now and into the future.
          Hide
          cjcowie Colvin Cowie added a comment - - edited

          Unfortunately I don't think that I personally can provide a unit test (without getting approval to do so).

          However I can reproduce the error reliably just by deploying the cloud example and creating a document:

          solr stop -all
          
          rm -rf ../example/cloud
          solr -e cloud -noprompt
          curl http://localhost:8983/solr/gettingstarted/update -d "[ {\"f_b\": true, \"id\": \"1\" }, {\"f_b\": false, \"id\": \"2\" }]"
          

          http://localhost:8983/solr/gettingstarted/select?facet.field=f_b&facet=on&indent=on&q=*:*&wt=json

          {
            "responseHeader":{
              "zkConnected":true,
              "status":0,
              "QTime":78,
              "params":{
                "q":"*:*",
                "facet.field":"f_b",
                "indent":"on",
                "facet":"on",
                "wt":"json"}},
            "response":{"numFound":2,"start":0,"maxScore":1.0,"docs":[
                {
                  "f_b":false,
                  "id":"1",
                  "_version_":1545007494796935168},
                {
                  "f_b":false,
                  "id":"2",
                  "_version_":1545007494796935168}]
            },
            "facet_counts":{
              "facet_queries":{},
              "facet_fields":{
                "f_b":[
                  "false",1,
                  "true",1]},
              "facet_ranges":{},
              "facet_intervals":{},
              "facet_heatmaps":{}}}
          

          The strange thing however, is that when I try and reproduce the problem with our actual configuration and test data which wrongly returns false on the javabin response, the json response returns the correct response. It appears in that case that toExternal() isn't being called at all. I don't know enough about the code to know why/how it would be bypassing toExternal() in that case though.

          Show
          cjcowie Colvin Cowie added a comment - - edited Unfortunately I don't think that I personally can provide a unit test (without getting approval to do so). However I can reproduce the error reliably just by deploying the cloud example and creating a document: solr stop -all rm -rf ../example/cloud solr -e cloud -noprompt curl http://localhost:8983/solr/gettingstarted/update -d "[ {\"f_b\": true, \"id\": \"1\" }, {\"f_b\": false, \"id\": \"2\" }]" http://localhost:8983/solr/gettingstarted/select?facet.field=f_b&facet=on&indent=on&q=*:*&wt=json { "responseHeader":{ "zkConnected":true, "status":0, "QTime":78, "params":{ "q":"*:*", "facet.field":"f_b", "indent":"on", "facet":"on", "wt":"json"}}, "response":{"numFound":2,"start":0,"maxScore":1.0,"docs":[ { "f_b":false, "id":"1", "_version_":1545007494796935168}, { "f_b":false, "id":"2", "_version_":1545007494796935168}] }, "facet_counts":{ "facet_queries":{}, "facet_fields":{ "f_b":[ "false",1, "true",1]}, "facet_ranges":{}, "facet_intervals":{}, "facet_heatmaps":{}}} The strange thing however, is that when I try and reproduce the problem with our actual configuration and test data which wrongly returns false on the javabin response, the json response returns the correct response. It appears in that case that toExternal() isn't being called at all. I don't know enough about the code to know why/how it would be bypassing toExternal() in that case though.
          Hide
          cjcowie Colvin Cowie added a comment - - edited

          I've spotted that the error with the json response happens when there is more than 1 shard in the collection.

          So toExternal() isn't being called when using a single shard and the json response writer, but it is used by the JavaBinCodec (via toObject() for BoolField) since javabin is used for communication between the shards?

          Show
          cjcowie Colvin Cowie added a comment - - edited I've spotted that the error with the json response happens when there is more than 1 shard in the collection. So toExternal() isn't being called when using a single shard and the json response writer, but it is used by the JavaBinCodec (via toObject() for BoolField) since javabin is used for communication between the shards?
          Hide
          danfox Dan Fox added a comment -

          Hi, I think I've also run into this issue. Here's a unit test that demonstrates it.

          Show
          danfox Dan Fox added a comment - Hi, I think I've also run into this issue. Here's a unit test that demonstrates it.
          Hide
          hossman Hoss Man added a comment -

          I've spotted that the error with the json response happens when there is more than 1 shard in the collection.

          So toExternal() isn't being called when using a single shard and the json response writer, but it is used by the JavaBinCodec (via toObject() for BoolField) since javabin is used for communication between the shards?

          Gah .. yes, of course, that's totally plausable. It should have occured to me when you posted your original JSON example that you might have bene using SolrCloud. That would explain why you saw it with JSON, but Pavan only reported seeing it using javabin, and not with JSON.

          (I assumed that since your reports conflicted, there must have been some other independent variable you had in common – but it was actaully a difference you two had: cloud mode)

          Dan: thank you for the patch, I've refactored the key bit into our existing SolrExampleTests, so it's run as part of many tests (embedded, Jetty with XML, jetty with javabin, etc...)

          With the attached test patch, all of these tests fail...

             [junit4] Tests with failures [seed: 4947C3B3180BA616]:
             [junit4]   - org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest.testExampleConfig
             [junit4]   - org.apache.solr.client.solrj.embedded.SolrExampleJettyTest.testExampleConfig
             [junit4]   - org.apache.solr.client.solrj.SolrExampleBinaryTest.testExampleConfig
             [junit4]   - org.apache.solr.client.solrj.embedded.SolrExampleEmbeddedTest.testExampleConfig
          

          ..working on fix, I suspect Colvin's earlier comments in SOLR-9187 are correct and it's a trivial fix.

          Show
          hossman Hoss Man added a comment - I've spotted that the error with the json response happens when there is more than 1 shard in the collection. So toExternal() isn't being called when using a single shard and the json response writer, but it is used by the JavaBinCodec (via toObject() for BoolField) since javabin is used for communication between the shards? Gah .. yes, of course, that's totally plausable. It should have occured to me when you posted your original JSON example that you might have bene using SolrCloud. That would explain why you saw it with JSON, but Pavan only reported seeing it using javabin, and not with JSON. (I assumed that since your reports conflicted, there must have been some other independent variable you had in common – but it was actaully a difference you two had: cloud mode) Dan: thank you for the patch, I've refactored the key bit into our existing SolrExampleTests, so it's run as part of many tests (embedded, Jetty with XML, jetty with javabin, etc...) With the attached test patch, all of these tests fail... [junit4] Tests with failures [seed: 4947C3B3180BA616]: [junit4] - org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest.testExampleConfig [junit4] - org.apache.solr.client.solrj.embedded.SolrExampleJettyTest.testExampleConfig [junit4] - org.apache.solr.client.solrj.SolrExampleBinaryTest.testExampleConfig [junit4] - org.apache.solr.client.solrj.embedded.SolrExampleEmbeddedTest.testExampleConfig ..working on fix, I suspect Colvin's earlier comments in SOLR-9187 are correct and it's a trivial fix.
          Hide
          dsmiley David Smiley added a comment -

          Not to be pedantic, but I believe this isn't about SolrCloud, rather about distributed-search (shards). Neither require the other.

          Show
          dsmiley David Smiley added a comment - Not to be pedantic, but I believe this isn't about SolrCloud, rather about distributed-search (shards). Neither require the other.
          Hide
          hossman Hoss Man added a comment -

          patch w/fix i'm currently testing

          Show
          hossman Hoss Man added a comment - patch w/fix i'm currently testing
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d59715f14bc180ebb9b2aef8ebbcb02103e9fcc8 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d59715f ]

          SOLR-9490: Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage

          (cherry picked from commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d59715f14bc180ebb9b2aef8ebbcb02103e9fcc8 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d59715f ] SOLR-9490 : Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage (cherry picked from commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e in lucene-solr's branch refs/heads/master from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=60ce8d7 ]

          SOLR-9490: Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage

          Show
          jira-bot ASF subversion and git services added a comment - Commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e in lucene-solr's branch refs/heads/master from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=60ce8d7 ] SOLR-9490 : Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5aedab619af9c65136b18dac46c493994465485b in lucene-solr's branch refs/heads/master from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5aedab6 ]

          SOLR-9490: Merge remote-tracking branch 'refs/remotes/origin/master'

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5aedab619af9c65136b18dac46c493994465485b in lucene-solr's branch refs/heads/master from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5aedab6 ] SOLR-9490 : Merge remote-tracking branch 'refs/remotes/origin/master'
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Re-opened to back-port to 6.2.1

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Re-opened to back-port to 6.2.1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d5e75a4d4165dd39febdc373258cae12ca0eedae in lucene-solr's branch refs/heads/branch_6_2 from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d5e75a4 ]

          SOLR-9490: Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage

          (cherry picked from commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e)

          (cherry picked from commit d59715f)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d5e75a4d4165dd39febdc373258cae12ca0eedae in lucene-solr's branch refs/heads/branch_6_2 from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d5e75a4 ] SOLR-9490 : Fixed bugs in BoolField that caused it to erroneously return "false" for all docs depending on usage (cherry picked from commit 60ce8d7c549ef90cd6aaa9297bf31aeb3dd3417e) (cherry picked from commit d59715f)
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Closing after 6.2.1 release

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.2.1 release
          Hide
          TimOwen Tim Owen added a comment -

          Just to add to this, if anyone was using 6.2.0 and doing document updates, this bug affected Atomic Updates and will have reset all boolean fields in the document to false when updating other fields of the document i.e. the actually-stored and indexed values are changed. We discovered this just recently and noticed some documents had lost their original boolean value, because we had been doing Atomic updates during the period we were running 6.2.0 and that had reset the values in the document itself. Even though we've now upgraded to 6.2.1 so the displayed values are shown correctly, the stored values have now been changed.

          Show
          TimOwen Tim Owen added a comment - Just to add to this, if anyone was using 6.2.0 and doing document updates, this bug affected Atomic Updates and will have reset all boolean fields in the document to false when updating other fields of the document i.e. the actually-stored and indexed values are changed. We discovered this just recently and noticed some documents had lost their original boolean value, because we had been doing Atomic updates during the period we were running 6.2.0 and that had reset the values in the document itself. Even though we've now upgraded to 6.2.1 so the displayed values are shown correctly, the stored values have now been changed.

            People

            • Assignee:
              hossman Hoss Man
              Reporter:
              hossman Hoss Man
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development