[ATLAS-3132] Data Patch Fx: Improve Data Patching Performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: 2.0.0, trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 12/Apr/19
• Created: 11/Apr/19 15:15
• Updated: 25/Sep/19 19:30
• Resolved: 19/Apr/19 18:06

Description

Background

The Java patch framework (now called data patching framework) introduced recently performs patching at the rate of 1 million entities per 15 hrs. This can be improved.

Proposed Solution

• Use the Producer-Consumer framework to spawn multiple workers to perform concurrent updates to entity vertices.
• Use AtlasGraph in bulk loading mode to further gain performance.
• Perform duplicate data checks during processing.

Projected Performance Improvement

• Based on various tests, these give increased throughput. New rate can be ~300K entities per 5 mins.