Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1594

KafkaWriter is asynchronous and may lose data on node failure

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Labels:
      None

      Description

      Currently, we do not block for data to be sent from kafka writer and we do not batch. Unfortunately, the send call is asynchronous, so if the node dies before the data is actually sent from kafka then it will drop data. It is highly unlikely that we will be able to make kafkawriter synchronous in a performant way, so we will likely need to batch in the parser and enrichment topologies.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mmiklavcic Michael Miklavcic
                Reporter:
                mmiklavcic Michael Miklavcic
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: