Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-2039

Multivalued fields with dynamic names does not work properly with DIH

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1
    • Fix Version/s: 6.3, master (7.0)
    • Labels:
      None
    • Environment:

      Windows XP, Default Solr 1.4 install with jetty

      Description

      Attempting to use multiValued fields using the DataImportHandler with dynamic names does not seem to work as intended.

      Setting up the following in schema.xml
      <dynamicField name="*_s" type="string" indexed="true" stored="true" multiValued="true" />

      Then attempting to import a multiValued field through a RDBMS using children records with a dynamic name. The snippet from data-config.xml

      <entity name="terms" query="select distinct DESC_TERM from tem_metadata where item_id=$

      {item.ID_PK}">
      <entity name="metadata" query="select * from term_metadata where item_id=${item.ID_PK}

      AND desc_term='$

      {terms.DESC_TERM}'" >
      <field name="${terms.DESC_TERM}

      _s" column="TEXT_VALUE" />
      </entity>
      </entity>

      Results in only the FIRST record being imported. If we change the field name in the above example to a constant (ie above field entry becomes: <field name="metadata_record_s" column="TEXT_VALUE" />). The multiple records are correctly imported.

      This was posted on solr-user, and others have reported the problem on the mailing list archive.

      The workaround was to use a javascript transform to perform the same behavior. Changes in data-config.xml become:

      <entity name="metadata" query="select * from vw_item_metadata where item_id=$

      {item.DIVID_PK}

      AND core_desc_term='$

      {terms.CORE_DESC_TERM}

      '" transformer="script:f1"/>

      <script><![CDATA[
      function f1(row)

      { var name = row.get('CORE_DESC_TERM'); var value = row.get('TEXT_VALUE') row.put(name+ '_s', value); return row; }

      ]]></script>

      This results in multivalued field with a dynamic name assigned.

      1. SOLR-2039.patch
        3 kB
        Cao Manh Dat
      2. SOLR-2039.patch
        0.8 kB
        ruslan.shv

        Activity

        Hide
        cmale Chris Male added a comment -

        +1 to fixing this. I have also seen this bug in the wild.

        Show
        cmale Chris Male added a comment - +1 to fixing this. I have also seen this bug in the wild.
        Hide
        koji Koji Sekiguchi added a comment -

        Just assigning a component

        Show
        koji Koji Sekiguchi added a comment - Just assigning a component
        Hide
        pavlickm ruslan.shv added a comment -

        how i understand, the issue is that DIH cannot import dynamic fields with specific field names. for example, i have following
        schema.xml

        <dynamicField name="*_s" type="string" indexed="true" stored="true" multiValued="true" />

        data-config.xml

        <entity name="items" query="select * from item">
        <field name="${items.item_name}_s" column="item_value" />
        </entity>

        DB with following content

        item_name item_value
        item1 value1
        item2 value2

        and i want to Solr indexed my data such way, that i would get something like this

        <doc>
        <str name="item1_s">
        value1
        </str>
        <str name="item2_s">
        value2
        </str>
        </doc>

        Show
        pavlickm ruslan.shv added a comment - how i understand, the issue is that DIH cannot import dynamic fields with specific field names. for example, i have following schema.xml <dynamicField name="*_s" type="string" indexed="true" stored="true" multiValued="true" /> data-config.xml <entity name="items" query="select * from item"> <field name="${items.item_name}_s" column="item_value" /> </entity> DB with following content item_name item_value item1 value1 item2 value2 and i want to Solr indexed my data such way, that i would get something like this <doc> <str name="item1_s"> value1 </str> <str name="item2_s"> value2 </str> </doc>
        Hide
        pavlickm ruslan.shv added a comment -

        I hope this patch will help solve the issue

        Show
        pavlickm ruslan.shv added a comment - I hope this patch will help solve the issue
        Hide
        caomanhdat Cao Manh Dat added a comment -

        An updated patch for this issues, along with a passed test on the lastest code.

        Show
        caomanhdat Cao Manh Dat added a comment - An updated patch for this issues, along with a passed test on the lastest code.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit b8d9647307c5559706aeec3aad32c2e416188979 in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b8d9647 ]

        SOLR-2039: Multivalued fields with dynamic names does not work properly with DIH

        Show
        jira-bot ASF subversion and git services added a comment - Commit b8d9647307c5559706aeec3aad32c2e416188979 in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b8d9647 ] SOLR-2039 : Multivalued fields with dynamic names does not work properly with DIH
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 279647a303750408d10f6f8a6c27a066176fe49e in lucene-solr's branch refs/heads/branch_6x from Shalin Shekhar Mangar
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=279647a ]

        SOLR-2039: Multivalued fields with dynamic names does not work properly with DIH

        (cherry picked from commit b8d9647)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 279647a303750408d10f6f8a6c27a066176fe49e in lucene-solr's branch refs/heads/branch_6x from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=279647a ] SOLR-2039 : Multivalued fields with dynamic names does not work properly with DIH (cherry picked from commit b8d9647)
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Thanks ruslan and Dat for the patches!

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Thanks ruslan and Dat for the patches!
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            shalinmangar Shalin Shekhar Mangar
            Reporter:
            harrysmithwla K A
          • Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development