Solr
  1. Solr
  2. SOLR-742

Unable to create dynamic fields with custom DataImportHandler transformer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Labels:
      None

      Description

      Discussion at: http://www.nabble.com/Creating-dynamic-fields-with-DataImportHandler-to19226532.html

      Dynamic fields aren't created when specified in a DataImportHandler's transformer.

      Reproducing the issue:
      I have defined a dynamic field (of type sdouble) in my schema called "_dynamic*". Inside the transformer's transformRow method, I am adding the name-value pair "_dynamicTest" and '1.0'. No errors are observed, but the data does not appear in the index after importing is complete.

      Interestingly, I can specify that same name-value pair combination in the DataImportHandler's config file, and it does appear in the index.

      1. SOLR-742.patch
        8 kB
        Shalin Shekhar Mangar
      2. SOLR-742.patch
        6 kB
        Noble Paul
      3. SOLR-742.patch
        10 kB
        Shalin Shekhar Mangar
      4. SOLR-742.patch
        8 kB
        Shalin Shekhar Mangar
      5. SOLR-742.patch
        3 kB
        Noble Paul
      6. SOLR-742.patch
        3 kB
        Noble Paul

        Activity

        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 709088.

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 709088.
        Hide
        Shalin Shekhar Mangar added a comment -

        We also need to keep a lowercase map of schema fields. The only caveat is that dynamic fields which are added by the transformers must be in the correct case as defined in schema.xml

        Show
        Shalin Shekhar Mangar added a comment - We also need to keep a lowercase map of schema fields. The only caveat is that dynamic fields which are added by the transformers must be in the correct case as defined in schema.xml
        Hide
        Noble Paul added a comment -

        simplified fix

        Show
        Noble Paul added a comment - simplified fix
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 708967.

        Thanks Wojtek and Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 708967. Thanks Wojtek and Noble!
        Hide
        Shalin Shekhar Mangar added a comment -
        1. Fixes a NPE when the rootEntity is null
        2. Added a test in TestDocBuilder2

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - Fixes a NPE when the rootEntity is null Added a test in TestDocBuilder2 I'll commit this shortly.
        Hide
        Shalin Shekhar Mangar added a comment -

        The last patch had a bug with handling nested entities.

        This patch simplifies the logic by iterating over the map instead of the Fields defined in data config. Also, the overhead of creating a lowercase-key map per row is removed.

        Show
        Shalin Shekhar Mangar added a comment - The last patch had a bug with handling nested entities. This patch simplifies the logic by iterating over the map instead of the Fields defined in data config. Also, the overhead of creating a lowercase-key map per row is removed.
        Hide
        Shalin Shekhar Mangar added a comment -

        The current DataImportHandler code does not handle dynamic fields. This patch solves the core problem partially by allowing Transformers to add a dynamic field/value pair to the document. Ideally, we'd like to support variables in the 'name' attribute as well.

        Once 1.3 is release, we should look at adding complete support for dynamic fields.

        Show
        Shalin Shekhar Mangar added a comment - The current DataImportHandler code does not handle dynamic fields. This patch solves the core problem partially by allowing Transformers to add a dynamic field/value pair to the document. Ideally, we'd like to support variables in the 'name' attribute as well. Once 1.3 is release, we should look at adding complete support for dynamic fields.
        Hide
        Wojtek Piaseczny added a comment -

        That fixed my issue, thank you!

        Show
        Wojtek Piaseczny added a comment - That fixed my issue, thank you!
        Hide
        Noble Paul added a comment -

        the old patch was wrong

        Show
        Noble Paul added a comment - the old patch was wrong
        Hide
        Noble Paul added a comment -

        dynamic fields are added if they are present in the doc

        Show
        Noble Paul added a comment - dynamic fields are added if they are present in the doc

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Wojtek Piaseczny
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development