Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3132

Data Patch Fx: Improve Data Patching Performance

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: 2.0.0, trunk
    • Component/s: atlas-core
    • Labels:
      None

      Description

      Background

      The Java patch framework (now called data patching framework) introduced recently performs patching at the rate of 1 million entities per 15 hrs. This can be improved.

      Proposed Solution

      • Use the Producer-Consumer framework to spawn multiple workers to perform concurrent updates to entity vertices.
      • Use¬†AtlasGraph in bulk loading mode to further gain performance.
      • Perform duplicate data checks during processing.

      Projected Performance Improvement

      • Based on various tests, these give increased throughput. New rate can be ~300K entities per 5 mins.

        Attachments

          Activity

            People

            • Assignee:
              amestry Ashutosh Mestry
              Reporter:
              amestry Ashutosh Mestry
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: