Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3878

Notifications: Improve Memory Usage in Scale Enviroment

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, trunk
    • Fix Version/s: trunk
    • Component/s: atlas-core
    • Labels:
      None

      Description

      Background

      As part of entity creation, Atlas sends notifications of different types. Current implementation, to listeners. Listeners in turn perform specific tasks.

      At a more concrete level, the EntityAuditListenerV2 will write audits and the NotificationEntityChangeListener will send Kafka notifications.

      Each of the listeners create notification objects. The notification objects are large in number and are short lived.

      The transient nature of the notification objects causes memory pressure in scale environment.

      Solution

      Create object pool for notification objects. This way objects can be reused.and existing design can be kept in tact. This will also offer benefit of using existing test setup for verification.

      Tests Used

      Setup 

      Create a test rig that will spawn multiple works that will invoke Atlas' bulk APIs for entity creation.

      Node: 40 workers, 8 GB allocated memory and 40 cores.

      Observation

      About 40 mins into the exercise, memory pressure builds up causing GC collects to take longer. This causes ZK timeout and finally Atlas process crashes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amestry Ashutosh Mestry
                Reporter:
                amestry Ashutosh Mestry
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: