Uploaded image for project: 'ActiveMQ Artemis'
  1. ActiveMQ Artemis
  2. ARTEMIS-2174

Broker reconnect to another with scale down policy cause OOM

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.3
    • Fix Version/s: 2.6.4, 2.7.0
    • Component/s: Broker
    • Labels:
      None

      Description

      When a node tries to reconnects to another node in a scale down cluster, the reconnect request gets denied by the other node and keeps retrying, which causes tasks in the ordered executor accumulate and eventually OOM.

      To reproduce:

      1. Start 2 nodes (node1 and 2) cluster configured in scale down mode.
      2. stop node2 and restart it.
      3. node1 will try to reconnect to node2 repeatedly and ever succeed.
      4. Inspect the connecting ClientSessionFactory (like adding log) and its threadpool (closeExecutor an object of OrderedExecutor) keeps adding tasks to its queue.

      Over the time the queue keeps ever growing, and will exhaust the heap memory.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gaohoward Howard Gao
                Reporter:
                gaohoward Howard Gao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: