Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1968

Messages are lost when a parser produces multiple messages and batch size is greater than 1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Labels:
      None

      Description

      A bug was discovered where messages are lost when a parser produces multiple messages.  This happens anytime the batch size for that sensor is set to greater than 1.  For example, consider a parser that produces 30 messages from a single input message.  Assume the batch size for this sensor/parser is set to 10.  The batch is currently flushed only after 10 tuples are received and only 10 messages are written out.  I think the correct behavior would be for 3 batches of 10 messages to be flushed for every tuple and a total of 300 messages written for every 10 tuples.

      This is happening because the various writer classes/interfaces (BulkWriterComponent, BulkMessageWriter, KafkaWriter, etc) assume a 1 to 1 relationship between messages and tuples.  The root cause of this specific issue is here.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rmerriman Ryan Merriman
                Reporter:
                rmerriman Ryan Merriman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 15h 20m
                  15h 20m