[ATLAS-4922] Atlas Parallel Import Functionality using Kafka

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: atlas-core
• Assignee: Disha Talreja
• Reporter: Disha Talreja
• Created: 15/Nov/24 18:27
• Updated: 15/Nov/24 18:27

Description

Develop an import process flow in Apache Atlas that leverages Kafka to manage parallel import requests. This solution will use a dedicated Kafka topic (ATLAS_IMPORT) to ensure import requests are processed efficiently and without conflicts. The ImportService will be enhanced to consume and process each message from the topic, enabling seamless behind-the-scenes import execution.

Users will be able to submit new import requests after the staging process, which prepares the file stream for Kafka, is complete. This approach supports concurrent imports and enhances the handling of import operations in Apache Atlas.

Additionally, this feature will introduce two new API endpoints for tracking the current status of imports. These endpoints will allow users to check both the overall progress and detailed status of each import request at any stage of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using the unique request ID. Users will also gain access to request metadata, providing insights into the status and progress of their imports.