Solr
  1. Solr
  2. SOLR-1001

using invariant request values from solrconfig.xml inside a data-config.xml regexp

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Labels:
      None

      Description

      As per several postings I noted that I can define variables inside an invariants list section of the DIH handler of solrconfig.xml. I can also reference these variables within data-config.xml. This works properly, the solr field "test" is nicely populated. However it is not substituted into my regex transformer? Here is my data-config.xml which gives a hint of the use case.

      <dataConfig>
      <dataSource name="myfilereader" type="FileDataSource"/>
      <document>
      <entity name="jc"
      processor="FileListEntityProcessor"
      fileName="^.*\.xml$"
      newerThan="'NOW-1000DAYS'"
      recursive="true"
      rootEntity="false"
      dataSource="null"
      baseDir="/Volumes/spare/ts/fords/dtd/fordsxml/data">
      <entity name="x"
      dataSource="myfilereader"
      processor="XPathEntityProcessor"
      url="$

      {jc.fileAbsolutePath}"
      stream="false"
      forEach="/record"
      transformer="DateFormatTransformer,TemplateTransformer,RegexTransformer,HTMLStripTransformer">

      <field column="fileAbsolutePath" template="${jc.fileAbsolutePath}

      " />
      <field column="fileWebPath" regex="$

      {dataimporter.request.finstalldir}(.*)" replaceWith="$1" sourceColName="fileAbsolutePath"/>
      <field column="test" template="${dataimporter.request.finstalldir}

      " />
      <field column="title" xpath="/record/title" />
      <field column="para" xpath="/record/sect1/para" stripHTML="true" />
      <field column="date" xpath="/record/metadata/date[@qualifier='Date']" dateTimeFormat="yyyyMMdd" />
      </entity>
      </entity>
      </document>
      </dataConfig>

      Shalin has pointed out that we are creating the regex Pattern without first resolving the variable. So we need to call VariableResolver.resolve on the 'regex' attribute's value before creating the Pattern object.

      1. SOLR-1001.patch
        4 kB
        Shalin Shekhar Mangar
      2. SOLR-1001.patch
        1 kB
        Noble Paul

        Activity

        Hide
        Fergus McMenemie added a comment -

        I could probably hack around this myself given Shalin's clue as to the cause. However two possible issues come to mind.

        • Is it possible that an equivalent change needs made to other transformers?
        • Is the construct $ {XXX}

          a valid part of a regular expression, I dont think so, but.... ?

        Show
        Fergus McMenemie added a comment - I could probably hack around this myself given Shalin's clue as to the cause. However two possible issues come to mind. Is it possible that an equivalent change needs made to other transformers? Is the construct $ {XXX} a valid part of a regular expression, I dont think so, but.... ?
        Hide
        Shalin Shekhar Mangar added a comment -

        Is it possible that an equivalent change needs made to other transformers?

        Yes, I think so. We need to review other transformer and entity processors to make sure we attempt to resolve all attributes.

        Is the construct ${XXX} a valid part of a regular expression, I dont think so, but.... ?

        Curly braces are used in regex to specify repetitions but they are not preceded by '$'. I think VariableResolver ignores any variables it is not able to resolve, so we should be fine; we will need to check this though.

        Show
        Shalin Shekhar Mangar added a comment - Is it possible that an equivalent change needs made to other transformers? Yes, I think so. We need to review other transformer and entity processors to make sure we attempt to resolve all attributes. Is the construct ${XXX} a valid part of a regular expression, I dont think so, but.... ? Curly braces are used in regex to specify repetitions but they are not preceded by '$'. I think VariableResolver ignores any variables it is not able to resolve, so we should be fine; we will need to check this though.
        Hide
        Noble Paul added a comment -

        this just fixes the RegexTransformer. We may take a look at the other Transformers and fix them too

        Show
        Noble Paul added a comment - this just fixes the RegexTransformer. We may take a look at the other Transformers and fix them too
        Hide
        Fergus McMenemie added a comment -

        Download and installed the patch. Works fine for me. Thanks very much.

        Show
        Fergus McMenemie added a comment - Download and installed the patch. Works fine for me. Thanks very much.
        Hide
        Shalin Shekhar Mangar added a comment -

        Would it be alright if we resolve all entity attributes in ContextImpl.getEntityAttribute? Similarily, we can change ContextImpl.getAllEntityFields to resolve field values just-in-time. This would keep this logic in one place and avoid this problem popping up in different places.

        Show
        Shalin Shekhar Mangar added a comment - Would it be alright if we resolve all entity attributes in ContextImpl.getEntityAttribute? Similarily, we can change ContextImpl.getAllEntityFields to resolve field values just-in-time. This would keep this logic in one place and avoid this problem popping up in different places.
        Hide
        Noble Paul added a comment -

        Would it be alright if we resolve all entity attributes in ContextImpl.getEntityAttribute?

        So the components will never be able to get the actual string if they wish to. Moreover it is not backcompat. We can add extra method

        1. ContextImpl.getResolvedEntityAttribute
        2. ContextImpl.getAllResolvedEntityFields .It is expensive to do that in here because the component may be interested in only one variable and we end up resolving all variables . If we can cache it it may be ok.
        Show
        Noble Paul added a comment - Would it be alright if we resolve all entity attributes in ContextImpl.getEntityAttribute? So the components will never be able to get the actual string if they wish to. Moreover it is not backcompat. We can add extra method ContextImpl.getResolvedEntityAttribute ContextImpl.getAllResolvedEntityFields .It is expensive to do that in here because the component may be interested in only one variable and we end up resolving all variables . If we can cache it it may be ok.
        Hide
        Shalin Shekhar Mangar added a comment -

        I see your point Noble. I don't really want to have a helper method in Context right now. Lets fix the issue at hand.

        I have made the following changes:

        1. HTMLStripTransformer allows variable in stripHTML attribute
        2. NumberFormatTransformer allows variable in formatStyle and locale attributes

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - I see your point Noble. I don't really want to have a helper method in Context right now. Lets fix the issue at hand. I have made the following changes: HTMLStripTransformer allows variable in stripHTML attribute NumberFormatTransformer allows variable in formatStyle and locale attributes I'll commit this shortly.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 741435.

        Thanks Fergus and Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 741435. Thanks Fergus and Noble!
        Hide
        Grant Ingersoll added a comment -

        Bulk close Solr 1.4 issues

        Show
        Grant Ingersoll added a comment - Bulk close Solr 1.4 issues

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Fergus McMenemie
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development