Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4397

PutElasticsearchHttp Processor: Elasticsearch URL property isn't expanding nifi language expressions

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Affects Version/s: 1.3.0
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
      None

      Description

      Despite the documentation saying that the PutElasticsearchHttp processor should support expression language this isn't happening. The Index property works correctly however. The property validates whether the value looks like a URL. But it seems like this validation is taking place before the expression is evaluated. Also, expressions like 'http://$

      {url}

      ' still are not evaluated.

      Technically I guess this isn't a 'major' issue, but it is quite painful to use it without the replacement happening. See also NIFI-4396 which is very similar behaviour.

      1. screenshot-1.png
        28 kB
        Joe Warner
      2. screenshot-2.png
        19 kB
        Joe Warner

        Activity

        Hide
        JPercivall Joseph Percivall added a comment -

        Hello Joe Warner, similar to my comment on NIFI-4396, can you elaborate a bit on what you mean by "expanding" the Nifi expressions and why you believe it's not working?

        Your comment "validation is taking place before the expression is evaluated" is correct and done on purpose. Validation of the properties is done after the processor configuration is saved and prior to running the processor. If expression language (EL) is present in a property then we can't fully validate it since a property may be reliant on an incoming FlowFiles' attributes or some other transient aspect to be evaluated. So for validation, we typically will first check if it contains EL, if so then mark it as valid. If it does not then continue on with proper validation.

        Show
        JPercivall Joseph Percivall added a comment - Hello Joe Warner , similar to my comment on NIFI-4396 , can you elaborate a bit on what you mean by "expanding" the Nifi expressions and why you believe it's not working? Your comment "validation is taking place before the expression is evaluated" is correct and done on purpose. Validation of the properties is done after the processor configuration is saved and prior to running the processor. If expression language (EL) is present in a property then we can't fully validate it since a property may be reliant on an incoming FlowFiles' attributes or some other transient aspect to be evaluated. So for validation, we typically will first check if it contains EL, if so then mark it as valid. If it does not then continue on with proper validation.
        Hide
        JoeWarner Joe Warner added a comment - - edited

        The behaviour I'm seeing now isn't what I remember, so my original description may not be accurate, but I think the following should illustrate the problem.

        I want to be able to set the 'Elasticsearch URL' property dynamically from an atttribute. Assume I have an attribute set as follows, es_host = http://<my_es_server>. Then if I set
        'Elasticsearch URL' = $

        {es_host} then what I get is



        I also tried it with es_host = <my_es_server> and 'Elasticsearch URL' = http://${es_host}

        and what I get is

        Show
        JoeWarner Joe Warner added a comment - - edited The behaviour I'm seeing now isn't what I remember, so my original description may not be accurate, but I think the following should illustrate the problem. I want to be able to set the 'Elasticsearch URL' property dynamically from an atttribute. Assume I have an attribute set as follows, es_host = http://<my_es_server>. Then if I set 'Elasticsearch URL' = $ {es_host} then what I get is I also tried it with es_host = <my_es_server> and 'Elasticsearch URL' = http://$ {es_host} and what I get is
        Hide
        JPercivall Joseph Percivall added a comment -

        Hello Joe Warner, unfortunately, PutElasticsearchHttp doesn't expose a FlowFile for evaluating attributes for the URL[1]. It is done this way because it does the processing in batches (default size of 100) and it would be essentially random which FlowFile would get picked to evaluate the attributes for. A ticket is created for it here[2]. For a bit more information, I had actually created a new one here[3] when I forgot there was already one open.

        I'm going to close this ticket since there isn't a problem with the PutElasticsearchHttp URL field expanding EL.

        [1] https://github.com/apache/nifi/blob/4e4d14f86ff21d2c7d20d9e180acee54a2ddf93b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L225
        [2] https://issues.apache.org/jira/browse/NIFI-1927
        [3] https://issues.apache.org/jira/browse/NIFI-4415

        Show
        JPercivall Joseph Percivall added a comment - Hello Joe Warner , unfortunately, PutElasticsearchHttp doesn't expose a FlowFile for evaluating attributes for the URL [1] . It is done this way because it does the processing in batches (default size of 100) and it would be essentially random which FlowFile would get picked to evaluate the attributes for. A ticket is created for it here [2] . For a bit more information, I had actually created a new one here [3] when I forgot there was already one open. I'm going to close this ticket since there isn't a problem with the PutElasticsearchHttp URL field expanding EL. [1] https://github.com/apache/nifi/blob/4e4d14f86ff21d2c7d20d9e180acee54a2ddf93b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L225 [2] https://issues.apache.org/jira/browse/NIFI-1927 [3] https://issues.apache.org/jira/browse/NIFI-4415

          People

          • Assignee:
            Unassigned
            Reporter:
            JoeWarner Joe Warner
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development