Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-742

Unable to create dynamic fields with custom DataImportHandler transformer

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Labels:
      None

      Description

      Discussion at: http://www.nabble.com/Creating-dynamic-fields-with-DataImportHandler-to19226532.html

      Dynamic fields aren't created when specified in a DataImportHandler's transformer.

      Reproducing the issue:
      I have defined a dynamic field (of type sdouble) in my schema called "_dynamic*". Inside the transformer's transformRow method, I am adding the name-value pair "_dynamicTest" and '1.0'. No errors are observed, but the data does not appear in the index after importing is complete.

      Interestingly, I can specify that same name-value pair combination in the DataImportHandler's config file, and it does appear in the index.

      1. SOLR-742.patch
        3 kB
        Noble Paul
      2. SOLR-742.patch
        3 kB
        Noble Paul
      3. SOLR-742.patch
        8 kB
        Shalin Shekhar Mangar
      4. SOLR-742.patch
        10 kB
        Shalin Shekhar Mangar
      5. SOLR-742.patch
        6 kB
        Noble Paul
      6. SOLR-742.patch
        8 kB
        Shalin Shekhar Mangar

        Activity

        Hide
        gsingers Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        gsingers Grant Ingersoll added a comment - Bulk close for Solr 1.4
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Committed revision 709088.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Committed revision 709088.
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        We also need to keep a lowercase map of schema fields. The only caveat is that dynamic fields which are added by the transformers must be in the correct case as defined in schema.xml

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - We also need to keep a lowercase map of schema fields. The only caveat is that dynamic fields which are added by the transformers must be in the correct case as defined in schema.xml
        Hide
        noble.paul Noble Paul added a comment -

        simplified fix

        Show
        noble.paul Noble Paul added a comment - simplified fix
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Committed revision 708967.

        Thanks Wojtek and Noble!

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Committed revision 708967. Thanks Wojtek and Noble!
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -
        1. Fixes a NPE when the rootEntity is null
        2. Added a test in TestDocBuilder2

        I'll commit this shortly.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Fixes a NPE when the rootEntity is null Added a test in TestDocBuilder2 I'll commit this shortly.
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        The last patch had a bug with handling nested entities.

        This patch simplifies the logic by iterating over the map instead of the Fields defined in data config. Also, the overhead of creating a lowercase-key map per row is removed.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - The last patch had a bug with handling nested entities. This patch simplifies the logic by iterating over the map instead of the Fields defined in data config. Also, the overhead of creating a lowercase-key map per row is removed.
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        The current DataImportHandler code does not handle dynamic fields. This patch solves the core problem partially by allowing Transformers to add a dynamic field/value pair to the document. Ideally, we'd like to support variables in the 'name' attribute as well.

        Once 1.3 is release, we should look at adding complete support for dynamic fields.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - The current DataImportHandler code does not handle dynamic fields. This patch solves the core problem partially by allowing Transformers to add a dynamic field/value pair to the document. Ideally, we'd like to support variables in the 'name' attribute as well. Once 1.3 is release, we should look at adding complete support for dynamic fields.
        Hide
        wojtekpia Wojtek Piaseczny added a comment -

        That fixed my issue, thank you!

        Show
        wojtekpia Wojtek Piaseczny added a comment - That fixed my issue, thank you!
        Hide
        noble.paul Noble Paul added a comment -

        the old patch was wrong

        Show
        noble.paul Noble Paul added a comment - the old patch was wrong
        Hide
        noble.paul Noble Paul added a comment -

        dynamic fields are added if they are present in the doc

        Show
        noble.paul Noble Paul added a comment - dynamic fields are added if they are present in the doc

          People

          • Assignee:
            shalinmangar Shalin Shekhar Mangar
            Reporter:
            wojtekpia Wojtek Piaseczny
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development