[METRON-569] Enrichment topology duplicates messages

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.4.0
• Flags: Important
• Assignee: Unassigned
• Reporter: Domenic Puzio
• Created: 15/Nov/16 20:30
• Updated: 17/Jul/17 16:22
• Resolved: 06/Jun/17 13:18

Description

When running the 'enrichment' topology, I get duplicate message being indexed. For example, I put 100 messages into the 'enrichment' Kafka queue and I get 175 messages onto the 'indexing' Kafka queue. This happens when I am running the 'enrichment' topology with one or more enrichment bolt.

This is an acking issue within the JoinBolt class. When a message does not "complete" the join (like when it is the first message in a pair of message to get joined) it does not get acked. This means that this message will get replayed through Storm, causing message duplication further down the road and tons of additional overhead. Adding the correct acking resolves this problem.

I will add the PR for this shortly.

Issue Links

• links to: GitHub Pull Request #359
• links to: GitHub Pull Request #359
• links to: GitHub Pull Request #603