[ATLAS-4121] Import Service: Improve Speed of Ingest for replicatedTo Option

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Fix Version/s: trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Created: 01/Feb/21 18:08
• Updated: 02/Feb/21 22:57

Description

Background

For Import replication options that includes skipLineage and replicatedTo, __ current implementation uses one entity-at-time import approach.

This approach can be end up taking long time to import for large payloads.

Solution

• Use an approach that allows for entity of a type to be concurrently created. 
• This assumes that the incoming stream of entities to be imported is topologically sorted (where parent entities appear before the child entities).
• Create entities of a type.
• Once all entities of a type are created, fetch next type.