[ATLAS-3929] Kafka hook for Atlas to capture production/consumption

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: kafka-integration
• Assignee: Viktor Somogyi-Vass
• Reporter: Dennis Jaheruddin
• Created: 25/Aug/20 12:39
• Updated: 31/Aug/20 09:13

Description

Currently we already have hooks for many engines (such as Spark, Nifi, Flink). We also have a hook into the classical storage (HDFS). However, with streaming becoming more and more dominant, we should also get a hook into the streaming storage: Kafka.

Currently producers and consumers need to explicitly take atlas into account, but for stronger governance and auditability, it should be possible to capture every interaction with Kafka, regardless of how consumption or production is done.