[METRON-1968] Messages are lost when a parser produces multiple messages and batch size is greater than 1

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.7.1
• Assignee: Ryan Merriman
• Reporter: Ryan Merriman
• Created: 17/Jan/19 15:40
• Updated: 23/Apr/19 23:08
• Resolved: 23/Apr/19 20:57

Description

A bug was discovered where messages are lost when a parser produces multiple messages.  This happens anytime the batch size for that sensor is set to greater than 1.  For example, consider a parser that produces 30 messages from a single input message.  Assume the batch size for this sensor/parser is set to 10.  The batch is currently flushed only after 10 tuples are received and only 10 messages are written out.  I think the correct behavior would be for 3 batches of 10 messages to be flushed for every tuple and a total of 300 messages written for every 10 tuples.

This is happening because the various writer classes/interfaces (BulkWriterComponent, BulkMessageWriter, KafkaWriter, etc) assume a 1 to 1 relationship between messages and tuples.  The root cause of this specific issue is here.

Issue Links

• links to: GitHub Pull Request #1330