Uploaded image for project: 'Stratos'
  1. Stratos
  2. STRATOS-614

Scale down logic does not execute as intended

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Eventing
    • Labels:
      None

      Description

      According to a flow test carried out on the latest code base, scale down functionality does not work as intended.

      Scenario:
      1. Partition Max Instance Count set to 3
      2. Send a sample request load to take scale up action. Autoscaler will spin up 3 instances.
      3. Stop the sample request load.
      4. Let the service cluster scale down.
      5. One instance being terminated but two remains.

      As found in the Autoscaler logs Member Stat Context of the terminated member is not being removed from the Cluster Context. As a result the same member being selected in each scale down action round.

        Activity

        Hide
        imesh Imesh Gunaratne added a comment -

        This issue was fixed by introducing dedicated message queues for each message receiver instance.
        Changes were committed with commit revision: 06491aadae81b22699f57cfec007c4a4fa17295f

        Show
        imesh Imesh Gunaratne added a comment - This issue was fixed by introducing dedicated message queues for each message receiver instance. Changes were committed with commit revision: 06491aadae81b22699f57cfec007c4a4fa17295f
        Hide
        imesh Imesh Gunaratne added a comment -

        After doing some thorough tests the cause of the issue was identified:

        The problem was with the topology receiver queue, since the queue was singleton in Single JVM mode it was being shared among multiple topology event receivers.

        Show
        imesh Imesh Gunaratne added a comment - After doing some thorough tests the cause of the issue was identified: The problem was with the topology receiver queue, since the queue was singleton in Single JVM mode it was being shared among multiple topology event receivers.
        Hide
        nirmal Nirmal Fernando added a comment -

        AFAIS upon MemberTerminatedEvent it's getting removed (check
        MemberTerminatedEventListener).


        Best Regards,
        Nirmal

        Nirmal Fernando.
        PPMC Member & Committer of Apache Stratos,
        Senior Software Engineer, WSO2 Inc.

        Blog: http://nirmalfdo.blogspot.com/

        Show
        nirmal Nirmal Fernando added a comment - AFAIS upon MemberTerminatedEvent it's getting removed (check MemberTerminatedEventListener). – Best Regards, Nirmal Nirmal Fernando. PPMC Member & Committer of Apache Stratos, Senior Software Engineer, WSO2 Inc. Blog: http://nirmalfdo.blogspot.com/

          People

          • Assignee:
            imesh Imesh Gunaratne
            Reporter:
            imesh Imesh Gunaratne
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development