Solr
  1. Solr
  2. SOLR-8607

The Schema API refuses to add new fields that match existing dynamic fields

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 5.3.1
    • Fix Version/s: 5.5, 6.0
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      If the all catch dynamic field '*' is present in a managed schema collection, then any field creation requests through Schema API return the confusing/weird error "Field ${FIELD_NAME} already exists".

      Schema of a test collection:

      curl -X GET http://localhost:8983/solr/testSQLQueries-eventsim/schema
      {
        "responseHeader":{
          "status":0,
          "QTime":0},
        "schema":{
          "name":"example",
          "version":1.5,
          "uniqueKey":"id",
          "fieldTypes":[{
              "name":"boolean",
              "class":"solr.BoolField",
              "sortMissingLast":true},
            {
              "name":"double",
              "class":"solr.TrieDoubleField",
              "positionIncrementGap":"0",
              "precisionStep":"0"},
            {
              "name":"float",
              "class":"solr.TrieFloatField",
              "positionIncrementGap":"0",
              "precisionStep":"0"},
            {
              "name":"int",
              "class":"solr.TrieIntField",
              "positionIncrementGap":"0",
              "precisionStep":"0"},
            {
              "name":"long",
              "class":"solr.TrieLongField",
              "precisionStep":"6"},
            {
              "name":"string",
              "class":"solr.StrField",
              "sortMissingLast":true},
            {
              "name":"tdate",
              "class":"solr.TrieDateField",
              "positionIncrementGap":"0",
              "precisionStep":"6"}],
          "fields":[{
              "name":"_root_",
              "type":"string",
              "indexed":true,
              "stored":false},
            {
              "name":"_version_",
              "type":"long",
              "indexed":true,
              "stored":true},
            {
              "name":"id",
              "type":"string",
              "indexed":true,
              "required":true,
              "stored":true}],
          "dynamicFields":[{
              "name":"*_tdt",
              "type":"tdate",
              "indexed":true,
              "stored":true},
            {
              "name":"*_txt",
              "type":"string",
              "multiValued":true,
              "indexed":true,
              "stored":true},
            {
              "name":"*_ss",
              "type":"string",
              "multiValued":true,
              "indexed":true,
              "stored":true},
            {
              "name":"*_ii",
              "type":"int",
              "multiValued":true,
              "indexed":true,
              "stored":true},
            {
              "name":"*_ls",
              "type":"long",
              "multiValued":true,
              "indexed":true,
              "stored":true},
            {
              "name":"*_i",
              "type":"int",
              "indexed":true,
              "stored":true},
            {
              "name":"*_s",
              "type":"string",
              "indexed":true,
              "stored":true},
            {
              "name":"*_l",
              "type":"long",
              "indexed":true,
              "stored":true},
            {
              "name":"*_b",
              "type":"boolean",
              "indexed":true,
              "stored":true},
            {
              "name":"*_f",
              "type":"float",
              "indexed":true,
              "stored":true},
            {
              "name":"*_d",
              "type":"double",
              "indexed":true,
              "stored":true},
            {
              "name":"*",
              "type":"string",
              "indexed":true,
              "stored":true}],
          "copyFields":[]}}
      

      Create new field 'userId' using Schema API

      curl -X POST -H "Content-type:application/json" --data-binary '{"add-field": {"name":"userId","type":"string","indexed":"true","stored":"true"}}' http://localhost:8983/solr/testSQLQueries-eventsim/schema
      {
        "responseHeader":{
          "status":0,
          "QTime":6},
        "errors":[{
            "add-field":{
              "name":"userId",
              "type":"string",
              "indexed":"true",
              "stored":"true"},
            "errorMessages":["Field 'userId' already exists.\n"]}]}
      

      The field 'userId' does not exist in the schema but it is also not possible to add this field because of the dynamic field '*'.

      While one should never define dynamic field '*' in a managed schema, the Schema API should atleast return some useful error message.

      1. SOLR-8607.patch
        5 kB
        Steve Rowe
      2. SOLR-8607.patch
        5 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Steve Rowe added a comment - - edited

          I see this same problem for other dynamic fields, not just the catch all *.

          To reproduce:

          $ bin/solr -e schemaless
          $ curl http://localhost:8983/solr/gettingstarted/schema
          [...]
          "dynamicFields":[...,{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true},
          [...] 
          $ curl -X POST -H "Content-type:application/json" --data-binary '{"add-field": {"name":"new_txt","type":"string","indexed":"true","stored":"true"}}' http://localhost:8983/solr/gettingstarted/schema
          [...]
          "errorMessages":["Field 'new_txt' already exists.\n"]}]}
          
          Show
          Steve Rowe added a comment - - edited I see this same problem for other dynamic fields, not just the catch all * . To reproduce: $ bin/solr -e schemaless $ curl http://localhost:8983/solr/gettingstarted/schema [...] "dynamicFields":[...,{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true}, [...] $ curl -X POST -H "Content-type:application/json" --data-binary '{"add-field": {"name":"new_txt","type":"string","indexed":"true","stored":"true"}}' http://localhost:8983/solr/gettingstarted/schema [...] "errorMessages":["Field 'new_txt' already exists.\n"]}]}
          Hide
          Steve Rowe added a comment -

          This workaround allowed me to create the new_txt field: first delete the matching dynamic field, then add the new field, then re-add the deleted dynamic field, all in one request:

          $ curl -X POST -H "Content-type:application/json" --data-binary '{
          "delete-dynamic-field":{"name":"*_txt"},
          "add-field": {"name":"new_txt","type":"string","indexed":"true","stored":"true"},
          "add-dynamic-field":{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true}}' \
          http://localhost:8983/solr/gettingstarted/schema
          
          {"responseHeader": {"status":0, "QTime":81}}
          
          $ curl http://localhost:8983/solr/gettingstarted/schema
          [...]
          "fields":[...{"name":"new_txt", "type":"string", "indexed":true,"stored":true}],
          [...]
          "dynamicFields":[...,{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true},
          [...]
          
          Show
          Steve Rowe added a comment - This workaround allowed me to create the new_txt field: first delete the matching dynamic field, then add the new field, then re-add the deleted dynamic field, all in one request: $ curl -X POST -H "Content-type:application/json" --data-binary '{ "delete-dynamic-field":{"name":"*_txt"}, "add-field": {"name":"new_txt","type":"string","indexed":"true","stored":"true"}, "add-dynamic-field":{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true}}' \ http://localhost:8983/solr/gettingstarted/schema {"responseHeader": {"status":0, "QTime":81}} $ curl http://localhost:8983/solr/gettingstarted/schema [...] "fields":[...{"name":"new_txt", "type":"string", "indexed":true,"stored":true}], [...] "dynamicFields":[...,{"name":"*_txt", "type":"text_general", "indexed":true, "stored":true}, [...]
          Hide
          Erik Hatcher added a comment -

          SOLR-8624 looks like a duplicate issue - with a proposed fix.

          Show
          Erik Hatcher added a comment - SOLR-8624 looks like a duplicate issue - with a proposed fix.
          Hide
          Steve Rowe added a comment -

          Patch with fix suggested on SOLR-8624, including tests that fail without the fix and succeed with it.

          Running full test suite with the patch now.

          Show
          Steve Rowe added a comment - Patch with fix suggested on SOLR-8624 , including tests that fail without the fix and succeed with it. Running full test suite with the patch now.
          Hide
          Steve Rowe added a comment -

          Patch with CHANGES.txt entry.

          All Solr tests passed with the patch. Committing shortly.

          Show
          Steve Rowe added a comment - Patch with CHANGES.txt entry. All Solr tests passed with the patch. Committing shortly.
          Hide
          ASF subversion and git services added a comment -

          Commit 064c0ac00fed662183c4cb117f6aeb6d0f9fe1a1 in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=064c0ac ]

          SOLR-8607: The Schema API refuses to add new fields that match existing dynamic fields

          Show
          ASF subversion and git services added a comment - Commit 064c0ac00fed662183c4cb117f6aeb6d0f9fe1a1 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=064c0ac ] SOLR-8607 : The Schema API refuses to add new fields that match existing dynamic fields
          Hide
          ASF subversion and git services added a comment -

          Commit 2bcda57ffbcb1456c17580f36c83530e8ac44f65 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2bcda57 ]

          SOLR-8607: The Schema API refuses to add new fields that match existing dynamic fields (cherry-picked commit 064c0ac).

          Show
          ASF subversion and git services added a comment - Commit 2bcda57ffbcb1456c17580f36c83530e8ac44f65 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2bcda57 ] SOLR-8607 : The Schema API refuses to add new fields that match existing dynamic fields (cherry-picked commit 064c0ac).
          Hide
          Steve Rowe added a comment -

          Pushed to master and branch_5x.

          Thanks kiran and Jan Høydahl for reporting!

          Show
          Steve Rowe added a comment - Pushed to master and branch_5x. Thanks kiran and Jan Høydahl for reporting!
          Hide
          ASF subversion and git services added a comment -

          Commit 064c0ac00fed662183c4cb117f6aeb6d0f9fe1a1 in lucene-solr's branch refs/heads/master-solr-8621 from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=064c0ac ]

          SOLR-8607: The Schema API refuses to add new fields that match existing dynamic fields

          Show
          ASF subversion and git services added a comment - Commit 064c0ac00fed662183c4cb117f6aeb6d0f9fe1a1 in lucene-solr's branch refs/heads/master-solr-8621 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=064c0ac ] SOLR-8607 : The Schema API refuses to add new fields that match existing dynamic fields

            People

            • Assignee:
              Steve Rowe
              Reporter:
              kiran
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development