Solr
  1. Solr
  2. SOLR-181

Support for "Required" field Property

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: update
    • Labels:
      None

      Description

      In certain situations, it can be helpful to require every document in your index has a value for a given field. While ideally the indexing client(s) should be responsible enough to add all necessary fields, this patch allows it to be enforced in the Solr schema, by adding a required property to a field entry. For example, with this in the schema:

      <field name="name" type="nametext" indexed="true" stored="true" required="true"/>

      A request to index a document without a name field will result in this response:

      <result status="1">org.apache.solr.core.SolrException: missing required fields: name
      (and then, of course, the stack trace)
      </result>

      The meat of this patch is that DocumentBuilder.getDoc() throws a SolrException if not all required fields have values; this may not work well as is with SOLR-139, Support updateable/modifiable documents, and may have to be changed depending on that issue's final disposition.

      1. solr-181-required-fields.patch
        47 kB
        Ryan McKinley
      2. solr-181-required-fields.patch
        37 kB
        Greg Ludington

        Issue Links

          Activity

          Hide
          Greg Ludington added a comment -

          This patch allows a per-field "required" property that can be used to require clients to include certain fields to submit to the index.

          With this in the index:
          <field name="name" type="nametext" indexed="true" stored="true" required="true"/>

          A document submitted without a name field will throw a SolrException in DocumentBuilder.getDoc() resulting in this sort of message being sent back to the client:

          <result status="1">org.apache.solr.core.SolrException: missing required fields:
          name

          Issues with the patch as submitted:

          1) The rule is enforced, and the Exception thrown, from DocumentBuilder.getDoc(). Since this area may change in SOLR-139, this patch may need to be adjusted depending on the final result of SOLR-139.
          2) Fields with defaultValues are implicitly required, though currently this patch does not automatically make the uniqueKey field required. It may make sense to do this; however, there seems to be some debate on the mailing lists about this, so It is commented out with a //TODO for now. See SOLR-172.
          3) The RequiredFieldsTest case uses its own schema file, as otherwise all other tests would have to be retrofitted to add these required fields in their submissions, and all future test writers would have to keep this in mind, as well.

          Show
          Greg Ludington added a comment - This patch allows a per-field "required" property that can be used to require clients to include certain fields to submit to the index. With this in the index: <field name="name" type="nametext" indexed="true" stored="true" required="true"/> A document submitted without a name field will throw a SolrException in DocumentBuilder.getDoc() resulting in this sort of message being sent back to the client: <result status="1">org.apache.solr.core.SolrException: missing required fields: name Issues with the patch as submitted: 1) The rule is enforced, and the Exception thrown, from DocumentBuilder.getDoc(). Since this area may change in SOLR-139 , this patch may need to be adjusted depending on the final result of SOLR-139 . 2) Fields with defaultValues are implicitly required, though currently this patch does not automatically make the uniqueKey field required. It may make sense to do this; however, there seems to be some debate on the mailing lists about this, so It is commented out with a //TODO for now. See SOLR-172 . 3) The RequiredFieldsTest case uses its own schema file, as otherwise all other tests would have to be retrofitted to add these required fields in their submissions, and all future test writers would have to keep this in mind, as well.
          Hide
          Greg Ludington added a comment -

          SOLR-139 makes documents modifiable, instead of all-or-nothing added, which has implications for restricting fields.

          SOLR-172 covers similar ground specific to the uniqueKey field.

          Show
          Greg Ludington added a comment - SOLR-139 makes documents modifiable, instead of all-or-nothing added, which has implications for restricting fields. SOLR-172 covers similar ground specific to the uniqueKey field.
          Hide
          Ryan McKinley added a comment -

          Finally got a chance to look at this. It looks good. I made a few modifications:

          1. changed tabs to spaces
          2. Added javadoc comments to make it clear that RequiredFields must contain all fieldsWithDefaultValues
          3. The error now contains the documents uniqueKey
          4. moved the test to o.a.s.schema
          5. I added a non-final flag to SchemaField to say if the field is required.
          6. Modified IndexSchema.java to set the uniqueKey as required unless it is specified as required="false" in the schema
          7. Added required="true" to the example schema.xml
          8. Added required="false" to the test schema.xml (one test does not include it)

          As a note to anyone else looking at the change log, Greg's patch also modifies AbstractSolrTestCase and TestHarness to be able to check what status is expected from checkUpdateU

          I think this offers a good solution to the (mis)feature that you could have a null uniqueKey. This patch lets you have a null uniqueKey, but you have to configure it.

          Show
          Ryan McKinley added a comment - Finally got a chance to look at this. It looks good. I made a few modifications: 1. changed tabs to spaces 2. Added javadoc comments to make it clear that RequiredFields must contain all fieldsWithDefaultValues 3. The error now contains the documents uniqueKey 4. moved the test to o.a.s.schema 5. I added a non-final flag to SchemaField to say if the field is required. 6. Modified IndexSchema.java to set the uniqueKey as required unless it is specified as required="false" in the schema 7. Added required="true" to the example schema.xml 8. Added required="false" to the test schema.xml (one test does not include it) As a note to anyone else looking at the change log, Greg's patch also modifies AbstractSolrTestCase and TestHarness to be able to check what status is expected from checkUpdateU I think this offers a good solution to the (mis)feature that you could have a null uniqueKey. This patch lets you have a null uniqueKey, but you have to configure it.
          Hide
          Yonik Seeley added a comment -

          Haven't looked at the code, but the description looks fine.

          +1

          Show
          Yonik Seeley added a comment - Haven't looked at the code, but the description looks fine. +1
          Hide
          Ryan McKinley added a comment -

          commited

          Show
          Ryan McKinley added a comment - commited
          Hide
          Hoss Man added a comment -

          This bug was modified as part of a bulk update using the criteria...

          • Marked ("Resolved" or "Closed") and "Fixed"
          • Had no "Fix Version" versions
          • Was listed in the CHANGES.txt for 1.2

          The Fix Version for all 39 issues found was set to 1.2, email notification
          was suppressed to prevent excessive email.

          For a list of all the issues modified, search jira comments for this
          (hopefully) unique string: 20080415hossman2

          Show
          Hoss Man added a comment - This bug was modified as part of a bulk update using the criteria... Marked ("Resolved" or "Closed") and "Fixed" Had no "Fix Version" versions Was listed in the CHANGES.txt for 1.2 The Fix Version for all 39 issues found was set to 1.2, email notification was suppressed to prevent excessive email. For a list of all the issues modified, search jira comments for this (hopefully) unique string: 20080415hossman2

            People

            • Assignee:
              Ryan McKinley
              Reporter:
              Greg Ludington
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development