Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3431

Support batch update in Notify processor

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      NIFI-3216 added ability to wait for N signals. It supports waiting for N fragments split by SplitXXXX processors. However, since Notify processor has to increase count one by one by calling expensive replace cache operation over network, it doesn't provide a practical performance when user configured a flow looks like below as N glow:

      Split
       --> (original) -> Wait for N
       --> (split) -> Do something -> Notify 1
      

      This JIRA improves Notify processor by:

      • Add "Signal Buffer Count" to specify max number of flow files that can be buffered and update cache at once
      • Add "Signal Counter Delta" to specify delta grater than 1, EL supported

      "Signal Buffer Count" would be useful in a flow like this:

      Split
       --> (original) -> Wait for N
       --> (split) -> Filter or something -> Notify M
      

      Buffer incoming M flow files and perform cache replace operation once.

      So does "Signal Counter Delta":

      Split
       --> (original) -> Wait for N
       --> (split) -> Filter or something -> Merge M -> PutXXX -> Notify M
      

      Specify 'M' via Attribute Expression Language.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user pvillard31 commented on the issue:

          https://github.com/apache/nifi/pull/1466

          👍 full build with contrib-check, tested a workflow based on the example you provided in your excellent post. Thanks @ijokarumawak, merged to master.

          Show
          githubbot ASF GitHub Bot added a comment - Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1466 👍 full build with contrib-check, tested a workflow based on the example you provided in your excellent post. Thanks @ijokarumawak, merged to master.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/nifi/pull/1466

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1466
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a90fa9c285cce2beb1fea8fb266df402f8926fcf in nifi's branch refs/heads/master from Koji Kawamura
          [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a90fa9c ]

          NIFI-3431: Support batch update in Notify processor

          • Added Signal Counter Delta property
          • Added Signal Buffer Count property
          • Added processor property name and display name
          • Changed IOException handling from routing it to failure to throw
            RuntimeException, so that NiFi framework can yield the processor for a while and try again

          Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

          This closes #1466.

          Show
          jira-bot ASF subversion and git services added a comment - Commit a90fa9c285cce2beb1fea8fb266df402f8926fcf in nifi's branch refs/heads/master from Koji Kawamura [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a90fa9c ] NIFI-3431 : Support batch update in Notify processor Added Signal Counter Delta property Added Signal Buffer Count property Added processor property name and display name Changed IOException handling from routing it to failure to throw RuntimeException, so that NiFi framework can yield the processor for a while and try again Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #1466.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ijokarumawak commented on the issue:

          https://github.com/apache/nifi/pull/1466

          Dear reviewer, I wrote a blog post on how to use Wait/Notify processors which can take benefit from this PR. I hope this helps:
          http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

          Show
          githubbot ASF GitHub Bot added a comment - Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1466 Dear reviewer, I wrote a blog post on how to use Wait/Notify processors which can take benefit from this PR. I hope this helps: http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ijokarumawak opened a pull request:

          https://github.com/apache/nifi/pull/1466

          NIFI-3431: Support batch update in Notify processor

          • Added Signal Counter Delta property
          • Added Signal Buffer Count property
          • Changed IOException handling from routing it to failure to throw
            RuntimeException, so that NiFi framework can yield the processor for a while and try again

          Thank you for submitting a contribution to Apache NiFi.

          In order to streamline the review of the contribution we ask you
          to ensure the following steps have been taken:

              1. For all changes:
          • [x] Is there a JIRA ticket associated with this PR? Is it referenced
            in the commit message?
          • [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
          • [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
          • [x] Is your initial contribution a single, squashed commit?
              1. For code changes:
          • [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
          • [x] Have you written or updated unit tests to verify your changes?
          • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
          • [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
          • [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
              1. For documentation related changes:
          • [x] Have you ensured that format looks appropriate for the output in which it is rendered?
              1. Note:
                Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ijokarumawak/nifi nifi-3431

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/nifi/pull/1466.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1466


          commit c0e6190bb95d8ee701801942ec07cc2c2ee80923
          Author: Koji Kawamura <ijokarumawak@apache.org>
          Date: 2017-02-02T06:18:01Z

          NIFI-3431: Support batch update in Notify processor

          • Added Signal Counter Delta property
          • Added Signal Buffer Count property
          • Changed IOException handling from routing it to failure to throw
            RuntimeException, so that NiFi framework can yield the processor for a while and try again

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1466 NIFI-3431 : Support batch update in Notify processor Added Signal Counter Delta property Added Signal Buffer Count property Changed IOException handling from routing it to failure to throw RuntimeException, so that NiFi framework can yield the processor for a while and try again Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: For all changes: [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [x] Has your PR been rebased against the latest commit within the target branch (typically master)? [x] Is your initial contribution a single, squashed commit? For code changes: [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? [x] Have you written or updated unit tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? For documentation related changes: [x] Have you ensured that format looks appropriate for the output in which it is rendered? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-3431 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1466.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1466 commit c0e6190bb95d8ee701801942ec07cc2c2ee80923 Author: Koji Kawamura <ijokarumawak@apache.org> Date: 2017-02-02T06:18:01Z NIFI-3431 : Support batch update in Notify processor Added Signal Counter Delta property Added Signal Buffer Count property Changed IOException handling from routing it to failure to throw RuntimeException, so that NiFi framework can yield the processor for a while and try again

            People

            • Assignee:
              ijokarumawak Koji Kawamura
              Reporter:
              ijokarumawak Koji Kawamura
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development