[APEXMALHAR-1211] Evaluate usage of bloom filter for de-duplication

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: algorithms
• Assignee: Chaitanya Chebolu
• Reporter: Amol Kekre
• Created: 08/Jul/14 17:33
• Updated: 05/Nov/15 03:10

Description

We should look at providing a bloom filter for de-dup. That way the size of state would be manageable. This approach would not be 100% guarantee, but would be very high (six sigma?) and may be viable for a lot of use cases