[ATLAS-3797] Entity Creation: Refactoring: Improve Edge Creation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Created: 13/May/20 18:03
• Updated: 02/Jun/20 20:36
• Resolved: 18/May/20 04:54

Description

Background

ATLAS-3762 introduced edge creation improvement by improving on edge existence check.

The same change could be applied to another area within the implementation.

Investigation

This was discovered as part of volume testing done for high-volume import. This included importing over 100M entities. Various parts of the entity creation process had probes added to check for performance.

During this investigation the edge creation was doing the same existence check that was addressed in earlier fixes. Re-using the same implementation showed marked improvement in entity creation.

These were the parameters used for during import.

Node configuration: 128 GB RAM, 40 cores.

atlas.migration.mode.batch.size=1500
atlas.migration.mode.workers=40

Issue Links

• links to: Code Review