Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6762

New Processor - Watch Multiple S3 Buckets for content changes

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.9.2
    • Fix Version/s: None
    • Component/s: Extensions
    • Labels:

      Description

      Proposing New S3 processor that is a "variation" on the existing ListS3 Processor. For lack of better term, I will call this "Watch Multiple S3 Buckets Processor" (naming here is open to feedback). This new Processor takes an input attribute "Bucket" from the FlowFile, which could use NiFi ExpressionLanguage  to compute that value. Hence, the Processor could "watch" different buckets in an S3 Object Store.

      This processor must handle the State Management logic for multiple S3 buckets. The simplest solution for this problem add to modify the State Management logic to prepend the bucket name to this key/objectName that is being used to track state.

      Finally, the Dev should document the maximum number of buckets it will monitor at once, maximum amount of state/storage it can have to track the various listings in various buckets.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                aramCodez Aram S Openden
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 384h
                  384h
                  Remaining:
                  Remaining Estimate - 383.5h
                  383.5h
                  Logged:
                  Remaining Estimate - 383.5h
                  0.5h