Solr
  1. Solr
  2. SOLR-3347

deleteByQuery failing with SolrCloud without _version_ in schema.xml

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, Trunk
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Distributed execution of deleteByQuery("*:*") depends on the existence of a field _version_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

      I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.

      1. solrconfig.xml
        56 kB
        Benson Margulies
      2. schema.xml
        7 kB
        Benson Margulies
      3. provision-and-start.sh
        1 kB
        Benson Margulies
      4. 0001-Attempt-to-repro-problem-with-del-and-SolrCloud.patch
        24 kB
        Benson Margulies

        Issue Links

          Activity

          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.
          Hoss Man made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Hoss Man [ hossman ]
          Resolution Fixed [ 1 ]
          Hide
          Hoss Man added a comment -

          I'm going to go ahead and resolve this as fixed since the new sanity checks in SolrCloud mode should prevent this situation from ever happening (see linked issue).

          Show
          Hoss Man added a comment - I'm going to go ahead and resolve this as fixed since the new sanity checks in SolrCloud mode should prevent this situation from ever happening (see linked issue).
          Hide
          Hoss Man added a comment -

          As of SOLR-3745, SolrCloud will fail hard if you don't have _version_ in the schema.

          this can be relaxed if/when more fine grained pre-req checking is in place.

          Show
          Hoss Man added a comment - As of SOLR-3745 , SolrCloud will fail hard if you don't have _version_ in the schema. this can be relaxed if/when more fine grained pre-req checking is in place.
          Hoss Man made changes -
          Link This issue is related to SOLR-3745 [ SOLR-3745 ]
          Mark Miller made changes -
          Fix Version/s 5.0 [ 12321664 ]
          Robert Muir made changes -
          Fix Version/s 4.0 [ 12322551 ]
          Fix Version/s 4.0-BETA [ 12322455 ]
          Hide
          Robert Muir added a comment -

          rmuir20120906-bulk-40-change

          Show
          Robert Muir added a comment - rmuir20120906-bulk-40-change
          Hoss Man made changes -
          Fix Version/s 4.0 [ 12322455 ]
          Fix Version/s 4.0-ALPHA [ 12314992 ]
          Hide
          Hoss Man added a comment -

          bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

          Show
          Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
          Hoss Man made changes -
          Link This issue relates to SOLR-3432 [ SOLR-3432 ]
          Hide
          Jan Høydahl added a comment -

          Cool, Steven, it worked well with -p1. I updated http://wiki.apache.org/solr/HowToContribute with this info.

          Show
          Jan Høydahl added a comment - Cool, Steven, it worked well with -p1. I updated http://wiki.apache.org/solr/HowToContribute with this info.
          Hide
          Steve Rowe added a comment -

          git patches can be applied to svn checkouts with patch -p1, instead of the usual patch -p0 used with svn-generated patches.

          Show
          Steve Rowe added a comment - git patches can be applied to svn checkouts with patch -p1 , instead of the usual patch -p0 used with svn-generated patches.
          Hide
          Benson Margulies added a comment -

          Jan Høydahl,

          I don't normally do radical edits to the description. However, in this particular case, my initial diagnostic theory was so wildly wrong that I felt it appropriate to either edit the description or close the JIRA and open a new one.

          As for the patch file names, that's just laziness. On the other hand, I'm one of the many people around the ASF making use of the git mirrors, especially when working on code where I'm not a committer – thus 'git format-patch'. You're the first person around here to express any problem with that. I wonder if there's some trick to get git to make a patch that is indistinguishable from svn diff.

          Show
          Benson Margulies added a comment - Jan Høydahl, I don't normally do radical edits to the description. However, in this particular case, my initial diagnostic theory was so wildly wrong that I felt it appropriate to either edit the description or close the JIRA and open a new one. As for the patch file names, that's just laziness. On the other hand, I'm one of the many people around the ASF making use of the git mirrors, especially when working on code where I'm not a committer – thus 'git format-patch'. You're the first person around here to express any problem with that. I wonder if there's some trick to get git to make a patch that is indistinguishable from svn diff.
          Mark Miller made changes -
          Fix Version/s 4.0 [ 12314992 ]
          Benson Margulies made changes -
          Description Distributed execution of deleteByQuery("\*:\*") depends on the existence of a field _version_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

          I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.



          Distributed execution of deleteByQuery("\*:\*") depends on the existence of a field \_version\_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

          I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.



          Benson Margulies made changes -
          Description Distributed execution of deleteByQuery("*:*") depends on the existence of a field _version_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

          I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.



          Distributed execution of deleteByQuery("\*:\*") depends on the existence of a field _version_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

          I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.



          Benson Margulies made changes -
          Summary deleteByQuery failing with SolrCloud deleteByQuery failing with SolrCloud without _version_ in schema.xml
          Description I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          I can now attach a 'freeze-dried' example of this problem. The root of the tarball has a shell script: start-cloud.sh:

          {noformat}
          sh start-cloud.sh 8983 3
          {noformat}

          You can then observe some documents in the index.

          If you then run:

          {noformat}
          sh del.sh http://localhost:8983/solr/update
          {noformat}

          you will observe, I hope, that the documents are still there.

          The solrconfig.xml in the three shards (and the zoo_data, by extension) is an exact copy of that from the 'example' directory. The schema.xml is mine.

          The file is too big to attach, so I am copying it to people.apache.org:~bimargulies/del-test-case.tgz.



          Distributed execution of deleteByQuery("*:*") depends on the existence of a field _version_ in the schema. The default schema has no comment on this field to indicate its important or relevance to SolrCloud, and no message is logged nor error status returned when there is no such field. The code in DistributedUpdateProcessor just has an if statement that never ever does any local deleting without it.

          I don't know whether the intention was that this should work or not. If someone would clue me in, I'd make a patch for schema.xml to add comments, or a patch to D-U-P to add logging. If it was supposed to work, I'm probably not qualified to make the fix to make it work.



          Hide
          Jan Høydahl added a comment -

          Benson, here are a few friendly suggestions for working with JIRA and patches (ref http://wiki.apache.org/solr/HowToContribute)

          Please use the JIRA "Comment" feature instead of editing the main description. The main description should be kept as a short problem description only, not involving solution...

          Also, any potential patch should be in SVN diff format from project root and named SOLR-NNNN.patch, i.e. SOLR-3347.patch. JIRA will keep track of which is the newest one.

          Show
          Jan Høydahl added a comment - Benson, here are a few friendly suggestions for working with JIRA and patches (ref http://wiki.apache.org/solr/HowToContribute ) Please use the JIRA "Comment" feature instead of editing the main description. The main description should be kept as a short problem description only, not involving solution... Also, any potential patch should be in SVN diff format from project root and named SOLR-NNNN.patch, i.e. SOLR-3347 .patch. JIRA will keep track of which is the newest one.
          Benson Margulies made changes -
          Description I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          I can now attach a 'freeze-dried' example of this problem. The root of the tarball has a shell script: start-cloud.sh:

          {noformat}
          sh start-cloud.sh 8983 3
          {noformat}

          You can then observe some documents in the index.

          If you then run:

          {noformat}
          sh del.sh http://localhost:8983/solr/update
          {noformat}

          you will observe, I hope, that the documents are still there.

          The solrconfig.xml in the three shards (and the zoo_data, by extension) is an exact copy of that from the 'example' directory. The schema.xml is mine.



          I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          I can now attach a 'freeze-dried' example of this problem. The root of the tarball has a shell script: start-cloud.sh:

          {noformat}
          sh start-cloud.sh 8983 3
          {noformat}

          You can then observe some documents in the index.

          If you then run:

          {noformat}
          sh del.sh http://localhost:8983/solr/update
          {noformat}

          you will observe, I hope, that the documents are still there.

          The solrconfig.xml in the three shards (and the zoo_data, by extension) is an exact copy of that from the 'example' directory. The schema.xml is mine.

          The file is too big to attach, so I am copying it to people.apache.org:~bimargulies/del-test-case.tgz.



          Benson Margulies made changes -
          Description I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          Another is the solrconfig.xml that fails, and a third is the shell script I use to launch.

          I'm continuing to work on this; I guess the next step is to allow you to repro what I'm doing by replacing my private URP with a dummy and seeing if I can get the same wrong results.
          I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          I can now attach a 'freeze-dried' example of this problem. The root of the tarball has a shell script: start-cloud.sh:

          {noformat}
          sh start-cloud.sh 8983 3
          {noformat}

          You can then observe some documents in the index.

          If you then run:

          {noformat}
          sh del.sh http://localhost:8983/solr/update
          {noformat}

          you will observe, I hope, that the documents are still there.

          The solrconfig.xml in the three shards (and the zoo_data, by extension) is an exact copy of that from the 'example' directory. The schema.xml is mine.



          Benson Margulies made changes -
          Description I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          Another is the solrconfig.xml that fails, and a third is the shell script I use to launch.

          I'm continuing to work on this; I guess the next step is to allow you to repro what I'm doing by replacing my private URP with a dummy and seeing if I can get the same wrong results.
          I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          Since I typed this up the first time, I tried using a completely stock solrconfig.xml; I see the same problem. I've also used my provisioning script to launch a vanilla set of 'example' shards, that also works. So my attention is drawn, by seeming process of elimination, to my schema.xml, which is also attached.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          Another is the solrconfig.xml that fails, and a third is the shell script I use to launch.

          I'm continuing to work on this; I guess the next step is to allow you to repro what I'm doing by replacing my private URP with a dummy and seeing if I can get the same wrong results.
          Benson Margulies made changes -
          Attachment schema.xml [ 12522226 ]
          Benson Margulies made changes -
          Description I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave. In the debugger, the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          Another is the solrconfig.xml that fails, and a third is the shell script I use to launch.

          I'm continuing to work on this; I guess the next step is to allow you to repro what I'm doing by replacing my private URP with a dummy and seeing if I can get the same wrong results.
          I've got a SolrCloud configuration in which calling deleteByQuery on CloudSolrServer never works. The log looks plausible, but the documents never leave.

          I've watched the process in the debugger when just using post.jar on simple XML files with the delete on *:* and the commit, and the dbqlevel in DistributedUpdateProcessor never gets to 3.

          I'm going to attach a number of things. One is a unit test that passes, sadly. Someone might like some small improvements in there in the cloud test classes, even if you don't bother with the test itself.

          Another is the solrconfig.xml that fails, and a third is the shell script I use to launch.

          I'm continuing to work on this; I guess the next step is to allow you to repro what I'm doing by replacing my private URP with a dummy and seeing if I can get the same wrong results.
          Benson Margulies made changes -
          Attachment solrconfig.xml [ 12522190 ]
          Attachment provision-and-start.sh [ 12522191 ]
          Benson Margulies made changes -
          Field Original Value New Value
          Attachment 0001-Attempt-to-repro-problem-with-del-and-SolrCloud.patch [ 12522189 ]
          Hide
          Benson Margulies added a comment -

          A patch as explained.

          Show
          Benson Margulies added a comment - A patch as explained.
          Benson Margulies created issue -

            People

            • Assignee:
              Hoss Man
              Reporter:
              Benson Margulies
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development