Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-56

Create unified enrichment topology

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Currently each sensor requires it's own end-to-end topology. This improvement will separate parsing and enrichment into different topologies. There will be separate parsing topologies for each sensor that read from a sensor Kafka topic, parse messages and write back out to a common enrichment Kafka topic. The enrichment topology will read from the common enrichment topic and enrich messages based on sensor metadata.

      This approach offers several benefits:

      • Will require fewer Storm workers
      • Parser topologies can be added and removed without affecting the enrichment topology
      • Configuration will be less verbose and more intuitive

        Attachments

          Activity

            People

            • Assignee:
              rmerriman Ryan Merriman
              Reporter:
              rmerriman Ryan Merriman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: