[ATLAS-3641] Import Service: Support New 'zipDirect' Format for Import

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Created: 28/Feb/20 21:42
• Updated: 02/Mar/20 18:30
• Resolved: 02/Mar/20 18:30

Description

Background

Existing exported ZIP formats stores the entities and the order of import in 2 separate files. During import it is thus necessary for both these (entities and order) to be in memory to be able to correctly import.

This causes large import files not function well. Also it puts a lot of pressure on memory.

Existing format also uses 1 zip entry per entity. This does not effectively compress the files.

 The new format addresses these short comings.

Solution

Create a format that has:

• Entities are stored in the order such that they are correctly imported.
• Lends itself to be able to read as a stream. This will reduce memory constraints.
• Store entities as 1 zip entry. This will improve compression rate.

Issue Links

• links to: Code Review