Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-5593

Affinity change message leak on massive topology updates

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.7
    • 2.1
    • cache
    • None

    Description

      When late affinity assignment is enabled, we complete the exchange future with custom discovery event. Since discovery topology events usually are much faster than exchange futures completion, it is possible that a newly joined node can 'see' the affinity change messages that are related to previous topology versions when this node even was not present in the topology.

      When this message is received, an exchange future is created and this message is added to discoEvts list. However, this future never completes on this node because init() is never called. This means that this exchange future sits in the exchange set with the affinity change message.

      Since the number of topology changes (and, thus, messages) can be quite large, this leads to excessive memory consumption on the starting node. I've observed ~3Gb of heap wasted on one of the nodes when > 200 nodes were restarted.

      Attachments

        Issue Links

          Activity

            People

              amashenkov Andrey Mashenkov
              agoncharuk Alexey Goncharuk
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: