Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6762

New Processor - Watch Multiple S3 Buckets for content changes

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.9.2
    • None
    • Extensions

    Description

      Proposing New S3 processor that is a "variation" on the existing ListS3 Processor. For lack of better term, I will call this "Watch Multiple S3 Buckets Processor" (naming here is open to feedback). This new Processor takes an input attribute "Bucket" from the FlowFile, which could use NiFi ExpressionLanguageĀ  to compute that value. Hence, the Processor could "watch" different buckets in an S3 Object Store.

      This processor must handle the State Management logic for multiple S3 buckets. The simplest solution for this problem add to modify the State Management logic to prepend the bucket name to this key/objectName that is being used to track state.

      Finally, the Dev should document the maximum number of buckets it will monitor at once, maximum amount of state/storage it can have to track the various listings in various buckets.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              aramCodez Aram S Openden
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 384h
                  384h
                  Remaining:
                  Remaining Estimate - 383h
                  383h
                  Logged:
                  Remaining Estimate - 383h
                  1h