[IGNITE-13111] Simplify backward checking of node connection.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Labels: iep-45
• Assignee: Vladimir Steshin
• Reporter: Vladimir Steshin
• Created: 03/Jun/20 12:38
• Updated: 17/Jun/20 10:51
• Resolved: 17/Jun/20 09:54

Description

We should fix several drawbacks in the backward checking of failed node. They prolong node failure detection upto:
ServerImpl.CON_CHECK_INTERVAL + 2 * IgniteConfiguretion.failureDetectionTimeout + 300ms.

See:

• ‘NodeFailureResearch.patch'. It creates test 'FailureDetectionResearch' which emulates long answears on a failed node and measures failure detection delays.
• 'FailureDetectionResearch.txt' - results of the test.
• 'FailureDetectionResearch_fixed.txt' - results of the test after this fix.
• 'WostCaseStepByStep.txt' - description how the worst case happens.

Suggestion:

1) We can simplify backward connection checking as we implement IGNITE-13012. Once we get robust, predictable connection ping, we don't need to check previous node because we can see whether it sent ping to current node within failure detection timeout. If not, previous node can be considered lost.

Instead of:

// Node cannot connect to it's next (for local node it's previous).
                        // Need to check connectivity to it.
                        long rcvdTime = lastRingMsgReceivedTime;
                        long now = U.currentTimeMillis();

                        // We got message from previous in less than double connection check interval.
                        boolean ok = rcvdTime + effectiveExchangeTimeout() >= now;
                        TcpDiscoveryNode previous = null;

                        if (ok) {
                            // Check case when previous node suddenly died. This will speed up
                            // node failing.

                          Checking connection to previous node
                         }

we could wait for ping from previous node. Scenario:

• n1 (Node1) failed to connect to n2.
• n1 asks n3 to establish connection instead of n2.
• n3 waits for ping form n2 for the rest of failure detection timeout.
• If n3 received ping from n2, it connects with n1. Or answers n1 that n2 is considered alive.

2) Then, seems we can remove:

ServerImpl.SocketReader.isConnectionRefused(SocketAddress addr);

Issue Links

• is caused by: IGNITE-13016 Fix backward checking of failed node.
• is caused by: IGNITE-13014 Remove double checking of node availability.