Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
    • Flags:
      Important

      Description

      Add a new standard NiFi processor to supplement the DetectDuplicate processor. The difference is this one works at the record-level.

      DetectDuplicateRecord

      Caches records from each incoming FlowFile and determines if the cached record has already been seen. The name of user-defined properties determines the RecordPath values used to determine if a record is unique. If no user-defined properties are present, the entire record is used as the input to determine uniqueness. All duplicate records are routed to 'duplicate'. If the record is not determined to be a duplicate, the Processor routes the record to 'non-duplicate'.

      This processor makes two different filtering data structures available depending on the level of precision and amount of records the user wishes to process:

      • HashSet filter type will guarantee 100% duplicate detection at the expense of storing one hash per record.
      • BloomFilter filter type will use efficient/constant space through probabilistic guarantees. This is useful when processing an extremely large number of records and some false positives are acceptable (i.e. some records may be marked as duplicate even though they have not been seen before).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spiglitz Adam Fisher
                Reporter:
                spiglitz Adam Fisher
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 10m
                  6h 10m