[ATLAS-4464] Ingest: Improve Rate of Ingest

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: trunk
• Fix Version/s: trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Created: 28/Oct/21 20:42
• Updated: 02/Nov/21 17:53

Description

Background

Existing implementation of NotificationHookConsumer has linear complexity for ingestion. This has several impacts:

• Authorization policies will take longer to get enforced.
• Unpredictable wait times for metadata showing up in Atlas.

Solution

Implement a mechanism for processing messages such that:

• Determine dependencies within incoming messages.
• Dependent messages should get processed serially.
• Messages without dependencies are processed concurrently.

 

Issue Links

• links to: Code Review