Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-11555

Unable to await partitions release latch caused by coordinator failover

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8
    • Component/s: None
    • Labels:
      None

      Description

      Currently exchanges latches (both server and client) are deleted when the latch is completed. This leads to a hang in the following scenario:
      1) A grid with several nodes starts exchange latch sync
      2) All nodes send acks to coordinator
      3) Coordinator processes acks and sends final acks to some of the nodes
      4) These nodes process acks, complete and delete client latches
      5) Coordinator fails
      6) Nodes which did not receive final acks re-send the ack to a new coordinator
      7) Since the new coordinator already completed and deleted the client latch, it does not process re-sent ack correctly and only adds it to the pending messages.

      Looks like the root cause of this issue is latch deletion on final ack. We can safely delete the latch only when all nodes are guaranteed to process the messages. Luckily, since the latch is tied to the exchange process, we can safely delete the latch when the corresponding exchange completes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                agoncharuk Alexey Goncharuk
                Reporter:
                agoncharuk Alexey Goncharuk
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m