Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6998

PutAzureEventHub to utilize Batch, Properties & Partitioning key

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.11.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      Current implementation of PutAzureEventHub is too simplistic. It sends one flow file at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into the message's User defined properties section. And most important problem is that you can't specify the partitioning key which plays critical role in a delivery consistency across multiple partitions in case there are more partitions. Without partitioning key multiple use-cases can't choose NiFi as an option.

      Aim of this improvement is to consume up to specified amount of flow files, lift all attributes of each flow file into User defined properties section of each message and let processor to use one of those attributes as a partitioning key.

      Each flow file in a batch is sent asynchronously which is way more faster then processor wait for all confirmations (or errors) and based on the result messages are transfered to success or failure output.

        Attachments

        1. EH_01_flow.png
          67 kB
          Martin Šimek
        2. EH_02_flowFile.png
          36 kB
          Martin Šimek
        3. EH_03_processor_settings.png
          49 kB
          Martin Šimek
        4. EH_04_EventHub.png
          20 kB
          Martin Šimek
        5. EH_05_receivedMessage.png
          38 kB
          Martin Šimek

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cima Martin Šimek
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 72h
                  72h
                  Remaining:
                  Time Spent - 1h 40m Remaining Estimate - 70h 20m
                  70h 20m
                  Logged:
                  Time Spent - 1h 40m Remaining Estimate - 70h 20m
                  1h 40m