[APEXMALHAR-1071] Create stateless deduper - aka deduper which forgets its state upon failure or repartitioning

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.5
• Component/s: algorithms
• Labels: feature
• Assignee: Chandni Singh
• Reporter: Chetan Narsude
• Created: 30/Apr/14 01:57
• Updated: 03/Jan/16 10:00
• Resolved: 30/Apr/14 22:59

Description

Due to overload on HDFS, we need an operators which works reliably since the time it starts till it has failure. It's okay if the operator instance forgets the ids of the events it already processed upon restarts.