Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7197

In-place replacement in LookupRecord processor

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.12.0, 1.11.4
    • Extensions
    • None

    Description

      One of the promise of the record-oriented processors is to avoid any data splitting to increase overall performances. This JIRA is about improving the LookupRecord processor to support lookup replacement when the lookup key field is located into an array of the input data.

      Here is an example of data:

        {
          "locales": [
            {
              "language" : "fr",
              "region" : "CA"
            },
            {
              "language" : "fr",
              "region" : "FR"
            }
          ]
        }
      

      If we want to replace language for each occurrence in the array, we'd need to slip the data, do the lookup and then merge back everything together.

      Unfortunately we can't use something like:

      Result RecordPath: /locales[*]/language
      key: /locales[*]/language
      

      Because the provided result record path returns multiple fields.

      With this JIRA, I suggest to add a new boolean property for "in-place replacement" that would default to false to preserve current behavior. It would also allow to do multi replacements at one time.

      If true, the following requirements would be made:

      • the lookup service should be a single key lookup service
      • replacement will only support simple types

      Then, every dynamic property containing record path would be evaluated and the corresponding field will be used as the key with the lookup service and the field will be updated with the value returned by the lookup service.

      Example:

      [
        {
          "foo" : {
            "foo" : "key"
          },
          "locales": [
            {
              "language" : "fr",
              "region" : "CA"
            },
            {
              "language" : "fr",
              "region" : "FR"
            }
          ]
        }, 
          {
            "foo" : {
              "foo" : "key"
            },
            "locales": [
              {
                "language" : "fr",
                "region" : "CA"
              },
              {
                "language" : "fr",
                "region" : "FR"
              }
            ]
          }
      ]
      

      With a lookup service containing:

      fr => French
      CA => Canada
      FR => France
      key => value
      

      And the following dynamic properties:

      - lookupLanguage => /locales[*]/language
      - lookupRegion => /locales[*]/region
      - lookupFoo => /foo/foo
      

      should give

      [
        {
          "foo" : {
            "foo" : "value"
          },
          "locales": [
            {
              "language" : "French",
              "region" : "Canada"
            },
            {
              "language" : "French",
              "region" : "France"
            }
          ]
        },  {
            "foo" : {
              "foo" : "value"
            },
            "locales": [
              {
                "language" : "French",
                "region" : "Canada"
              },
              {
                "language" : "French",
                "region" : "France"
              }
            ]
          }
      ]
      

      Attachments

        Issue Links

          Activity

            People

              pvillard Pierre Villard
              pvillard Pierre Villard
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m