Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3716

Utility to calculate MAX storage required for BackPressure on a Node

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Today there is no way to understand the max amount of data which can be stored in backpressure for a given flow. Personally have had users configure queues with impossible backpressure amounts that would definitely fill the disk, having an operational tool to understand if anyone has done something so dramatic would be helpful.

      I see this produce a report by analyzing the FlowFile.xml.gz.

      • Total Storage for all queues Backpressure
      • Average Storage of all queues Backpressure
      • Min and Max of all queues Backpressure over the entire flow.

        Issue Links

          Activity

          Hide
          Eulicny Eric Ulicny added a comment -

          I want to work on this.

          Show
          Eulicny Eric Ulicny added a comment - I want to work on this.
          Hide
          aldrin Aldrin Piri added a comment -

          Hey Eric Ulicny,

          I've added you to the list of contributors in JIRA for the NIFI project and will get this ticket assigned to you. Thanks for getting involved!

          Show
          aldrin Aldrin Piri added a comment - Hey Eric Ulicny , I've added you to the list of contributors in JIRA for the NIFI project and will get this ticket assigned to you. Thanks for getting involved!
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user Xsixteen opened a pull request:

          https://github.com/apache/nifi/pull/1747

          NIFI-3716 Flowfile analyzer tool created

          Thank you for submitting a contribution to Apache NiFi.

          In order to streamline the review of the contribution we ask you
          to ensure the following steps have been taken:

              1. For all changes:
          • [ ] Is there a JIRA ticket associated with this PR? Is it referenced
            in the commit message?
          • [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
          • [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
          • [ ] Is your initial contribution a single, squashed commit?
              1. For code changes:
          • [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
          • [ ] Have you written or updated unit tests to verify your changes?
          • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
          • [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
          • [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
              1. For documentation related changes:
          • [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
              1. Note:
                Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/Xsixteen/nifi master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/nifi/pull/1747.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1747


          commit ff63415fe27ab4f64dc8149ec27cc162e31b567f
          Author: Eric <x16.tux@gmail.com>
          Date: 2017-05-01T01:33:12Z

          flowfile analyzer tool created


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user Xsixteen opened a pull request: https://github.com/apache/nifi/pull/1747 NIFI-3716 Flowfile analyzer tool created Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: For all changes: [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? [ ] Is your initial contribution a single, squashed commit? For code changes: [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? [ ] Have you written or updated unit tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? For documentation related changes: [ ] Have you ensured that format looks appropriate for the output in which it is rendered? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Xsixteen/nifi master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1747.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1747 commit ff63415fe27ab4f64dc8149ec27cc162e31b567f Author: Eric <x16.tux@gmail.com> Date: 2017-05-01T01:33:12Z flowfile analyzer tool created
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          I went and built this, you have alot of checkstyle issues -

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (check-style) on project nifi-toolkit-flowanalyzer: *You have 187 Checkstyle violations*. -> [Help 1]

          Mostly just white spaces, but you may wanna try building with:
          ```
          mvn -Pcontrib-check clean install
          ```

          That said I did built and run the tool. While it builds I am getting an error against an flowfile.
          ```
          /usr/jdk64/jdk1.8.0_60/bin/java -cp ./nifi-toolkit-flowanalyzer-1.2.0-SNAPSHOT.jar org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver /usr/hdf/nifi/conf/flow.xml.gz
          Using flowfile=/usr/hdf/nifi/conf/flow.xml.gz
          java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
          at java.math.BigDecimal.divide(BigDecimal.java:1690)
          at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:85)
          Exception Occured while attempting to parse Flowfile.xml. Cause: null
          ```

          Maybe rounding UP would be ok>?
          http://stackoverflow.com/questions/4591206/arithmeticexception-non-terminating-decimal-expansion-no-exact-representable

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 I went and built this, you have alot of checkstyle issues - [ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (check-style) on project nifi-toolkit-flowanalyzer: * You have 187 Checkstyle violations *. -> [Help 1] Mostly just white spaces, but you may wanna try building with: ``` mvn -Pcontrib-check clean install ``` That said I did built and run the tool. While it builds I am getting an error against an flowfile. ``` /usr/jdk64/jdk1.8.0_60/bin/java -cp ./nifi-toolkit-flowanalyzer-1.2.0-SNAPSHOT.jar org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver /usr/hdf/nifi/conf/flow.xml.gz Using flowfile=/usr/hdf/nifi/conf/flow.xml.gz java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result. at java.math.BigDecimal.divide(BigDecimal.java:1690) at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:85) Exception Occured while attempting to parse Flowfile.xml. Cause: null ``` Maybe rounding UP would be ok>? http://stackoverflow.com/questions/4591206/arithmeticexception-non-terminating-decimal-expansion-no-exact-representable
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Also would it be possible to change the packaging so that it builds its in a 'java jar' manor so the admin doesnt have to go lookup the entire classname to run the utility?

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 Also would it be possible to change the packaging so that it builds its in a 'java jar' manor so the admin doesnt have to go lookup the entire classname to run the utility?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          ignore my java -cp // java jar comments..

          Can you mimic what the other existing tools do here with .bin and .sh scripts ?

          https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 ignore my java -cp // java jar comments.. Can you mimic what the other existing tools do here with .bin and .sh scripts ? https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-assembly/src/main/resources/bin
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Test fails when building Looks trivial see below. Also built it with skipTests anyway just to run it again, on 1 Cluster I am getting a null.

          ```
          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.nifi.toolkit.FlowAnalyzerDriverTest
          Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec <<< FAILURE! - in org.apache.nifi.toolkit.FlowAnalyzerDriverTest
          convertBytesToGB(org.apache.nifi.toolkit.FlowAnalyzerDriverTest) Time elapsed: 0.006 sec <<< FAILURE!
          org.junit.ComparisonFailure: expected:<13[]> but was:<13[.000000000]>
          at org.junit.Assert.assertEquals(Assert.java:115)
          at org.junit.Assert.assertEquals(Assert.java:144)
          at org.apache.nifi.toolkit.FlowAnalyzerDriverTest.convertBytesToGB(FlowAnalyzerDriverTest.java:37)

          Results :

          Failed tests:
          FlowAnalyzerDriverTest.convertBytesToGB:37 expected:<13[]> but was:<13[.000000000]>

          ```

          ```
          /usr/jdk64/jdk1.8.0_60/bin/java -cp ./nifi-toolkit-flowanalyzer-1.2.0-SNAPSHOT.jar org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver /usr/hdf/nifi/conf/flow.xml.gz
          Using flowfile=/usr/hdf/nifi/conf/flow.xml.gz
          java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
          at java.math.BigDecimal.divide(BigDecimal.java:1690)
          at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:91)
          Exception Occured while attempting to parse Flowfile.xml. Cause: null
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 Test fails when building Looks trivial see below. Also built it with skipTests anyway just to run it again, on 1 Cluster I am getting a null. ``` ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.nifi.toolkit.FlowAnalyzerDriverTest Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec <<< FAILURE! - in org.apache.nifi.toolkit.FlowAnalyzerDriverTest convertBytesToGB(org.apache.nifi.toolkit.FlowAnalyzerDriverTest) Time elapsed: 0.006 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<13[]> but was:<13 [.000000000] > at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.nifi.toolkit.FlowAnalyzerDriverTest.convertBytesToGB(FlowAnalyzerDriverTest.java:37) Results : Failed tests: FlowAnalyzerDriverTest.convertBytesToGB:37 expected:<13[]> but was:<13 [.000000000] > ``` ``` /usr/jdk64/jdk1.8.0_60/bin/java -cp ./nifi-toolkit-flowanalyzer-1.2.0-SNAPSHOT.jar org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver /usr/hdf/nifi/conf/flow.xml.gz Using flowfile=/usr/hdf/nifi/conf/flow.xml.gz java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result. at java.math.BigDecimal.divide(BigDecimal.java:1690) at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:91) Exception Occured while attempting to parse Flowfile.xml. Cause: null ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Xsixteen commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Updated, please try again.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Xsixteen commented on the issue: https://github.com/apache/nifi/pull/1747 Updated, please try again.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Tests passed but still null. Attached my flow.xml.gz for you to work with. Also RAT went off about unlisc files

          ```
          Using flowfile=/data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz
          java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
          at java.math.BigDecimal.divide(BigDecimal.java:1690)
          at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:91)
          Exception Occured while attempting to parse Flowfile.xml. Cause: null
          ```
          [flow.xml.gz](https://github.com/apache/nifi/files/985009/flow.xml.gz)

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 Tests passed but still null. Attached my flow.xml.gz for you to work with. Also RAT went off about unlisc files ``` Using flowfile=/data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result. at java.math.BigDecimal.divide(BigDecimal.java:1690) at org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver.main(FlowAnalyzerDriver.java:91) Exception Occured while attempting to parse Flowfile.xml. Cause: null ``` [flow.xml.gz] ( https://github.com/apache/nifi/files/985009/flow.xml.gz )
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Xsixteen commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Fixed the issue with second Big Decimal division point. Added licensing information to project.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Xsixteen commented on the issue: https://github.com/apache/nifi/pull/1747 Fixed the issue with second Big Decimal division point. Added licensing information to project.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          works great now on multiple differing flow.xml.gz's... I am thinking that perhaps we should include the min/max/avg flow files per queue as well just in case some users decides to set it to 1billion for a 60 byte file...

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 works great now on multiple differing flow.xml.gz's... I am thinking that perhaps we should include the min/max/avg flow files per queue as well just in case some users decides to set it to 1billion for a 60 byte file...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          You need to update the root nifi project pom file, and the toolkit-assembly pom to include this otherwise it does not get packed into the toolkit assembly. When this happens your script exists but it cant find the libs because they didnt get properly packed into the assembly.

          when running the below we can probably change the message -
          ```
          >sh flow-analyzer.sh
          Usage: java -cp <jarfile> org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver <path to flowfile.xml.gz>
          ``

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 You need to update the root nifi project pom file, and the toolkit-assembly pom to include this otherwise it does not get packed into the toolkit assembly. When this happens your script exists but it cant find the libs because they didnt get properly packed into the assembly. when running the below we can probably change the message - ``` >sh flow-analyzer.sh Usage: java -cp <jarfile> org.apache.nifi.toolkit.flowanalyzer.FlowAnalyzerDriver <path to flowfile.xml.gz> ``
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user GrayGwizdz commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Reading through the code everything looks good, I always err on the side of the more test cases the better but I think you have your bases covered.

          Show
          githubbot ASF GitHub Bot added a comment - Github user GrayGwizdz commented on the issue: https://github.com/apache/nifi/pull/1747 Reading through the code everything looks good, I always err on the side of the more test cases the better but I think you have your bases covered.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Xsixteen commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Joe, I added functionality to list max/min/avg for <maxWorkQueueSize>.

          Gray - I completely agree, I will add more testing if complexity increases.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Xsixteen commented on the issue: https://github.com/apache/nifi/pull/1747 Joe, I added functionality to list max/min/avg for <maxWorkQueueSize>. Gray - I completely agree, I will add more testing if complexity increases.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          I think you just need a entry in your pom in the toolkit-assembly now and ull be all done

          Looks like it works well. Will test on some other clusters just to validate diversity.

          ```
          $ sh flow-analyzer.sh /data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz
          Using flowfile=/data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz
          Total Bytes Utilized by System=11 GB
          Max Back Pressure Size=2 GB
          Min Back Pressure Size=1 GB
          Average Back Pressure Size=1.222222222 GB
          Max Flowfile Queue Size=2147483647
          Min Flowfile Queue Size=10000
          Avg Flowfile Queue Size=477230810.444444444
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 I think you just need a entry in your pom in the toolkit-assembly now and ull be all done Looks like it works well. Will test on some other clusters just to validate diversity. ``` $ sh flow-analyzer.sh /data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz Using flowfile=/data/ssd01/builds/nifi-rc-120/nifi-1.2.0/conf/flow.xml.gz Total Bytes Utilized by System=11 GB Max Back Pressure Size=2 GB Min Back Pressure Size=1 GB Average Back Pressure Size=1.222222222 GB Max Flowfile Queue Size=2147483647 Min Flowfile Queue Size=10000 Avg Flowfile Queue Size=477230810.444444444 ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user josephxsxn commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Builds fine passed check-style. Tests pass. License headers exist. Tested on my custom workflow with success.

          Please rebase master from upstream and squash your 11 commits into 1 for the pull.

          Show
          githubbot ASF GitHub Bot added a comment - Github user josephxsxn commented on the issue: https://github.com/apache/nifi/pull/1747 Builds fine passed check-style. Tests pass. License headers exist. Tested on my custom workflow with success. Please rebase master from upstream and squash your 11 commits into 1 for the pull.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Xsixteen commented on the issue:

          https://github.com/apache/nifi/pull/1747

          I have successfully squashed the commits.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Xsixteen commented on the issue: https://github.com/apache/nifi/pull/1747 I have successfully squashed the commits.
          Hide
          josephxsxn Joseph Niemiec added a comment -

          Build, Tested, Squashed, Looks pretty good. Just needs a version pom update.

          Show
          josephxsxn Joseph Niemiec added a comment - Build, Tested, Squashed, Looks pretty good. Just needs a version pom update.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Xsixteen commented on the issue:

          https://github.com/apache/nifi/pull/1747

          Updated to SNAPSHOT version 1.3.0, then rebased.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Xsixteen commented on the issue: https://github.com/apache/nifi/pull/1747 Updated to SNAPSHOT version 1.3.0, then rebased.
          Hide
          joewitt Joseph Witt added a comment -

          Will take a look and get it reviewed. Thanks

          Show
          joewitt Joseph Witt added a comment - Will take a look and get it reviewed. Thanks

            People

            • Assignee:
              Eulicny Eric Ulicny
              Reporter:
              josephxsxn Joseph Niemiec
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development