[ATLAS-4323] Add GUID key to notification events

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: atlas-core
• Assignee: Unassigned
• Reporter: Adam Bellemare
• Created: 02/Jun/21 14:21
• Updated: 14/Jun/21 14:26

Description

Currently, Atlas event notifications written to Kafka do not have a key. There are several downsides to this:

1) Data for the same GUID will end up in different Kafka topic partitions (as the events will be written randomly, or using round-robin). There is no data locality at the partition level.

2) Consumers may consume events for the same GUID out-of-order. For example, it may consume all of the events in one partition before moving on to the next partition (round-robin consumption). Due to the lack of data locality in #1, there is no guarantee around the ordering in which a consumer will see events between partitions. This can result in inconsistent consumer behaviour or materialized state.

3) Unable to apply Topic Compaction. Kafka allows you to compact topics such that only the most recent event of a given key is maintained. By using a unique key, we can use Kafka compaction instead of using time-based expiry.

 

The GUID is unique for each entity and relationship, and is the ideal key for notification events. The default hash partitioner of the producer will ensure that all events of the same GUID are delivered to the same topic partition, remedying all 3 issues discussed above.