[ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0, 3.0.0
• Component/s: atlas-core
• Assignee: Saqeeb Shaikh
• Reporter: Madhan Neethiraj
• Created: 02/Jul/19 20:05
• Updated: 07/Jan/20 09:27
• Resolved: 13/Jul/19 17:42

Description

Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.

Issue Links

• duplicates: ATLAS-3169 Create priority Queue for Atlas messages coming from 2 different sources;