[METRON-1594] KafkaWriter is asynchronous and may lose data on node failure

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.6.0
• Assignee: Michael Miklavcic
• Reporter: Michael Miklavcic
• Created: 31/May/18 20:38
• Updated: 21/May/19 20:18
• Resolved: 06/Jun/18 21:41

Description

Currently, we do not block for data to be sent from kafka writer and we do not batch. Unfortunately, the send call is asynchronous, so if the node dies before the data is actually sent from kafka then it will drop data. It is highly unlikely that we will be able to make kafkawriter synchronous in a performant way, so we will likely need to batch in the parser and enrichment topologies.

Issue Links

• links to: GitHub Pull Request #1045