[RATIS-459] Async requests may become out-of-order in some rare case

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3.0
• Component/s: client, gRPC, server
• Labels: ozone
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 19/Dec/18 03:33
• Updated: 14/Mar/19 22:29
• Resolved: 14/Mar/19 19:08

Description

With some discussion with Lokesh Jain (thanks!), the following (highly unlikely) hypothetical scenario may lead to the async requests out-of-order.

1. Requests r1-r5 success,
2. Request r6 fails with NotLeader,
3. Request r7 fails with some other IOException
4. r6 is still keeping retrying and a retry of r7 succeeds with s1.

Below are the suggested fix:

• In SlidingWindow.Client, resetFirstSeqNum() should, but currently does not, clear the request map.
• If a request is replied with NotLeaderException, it should not be removed from the request map until resetFirstSeqNum() above.

Issue Links

• is related to: RATIS-499 Leader should close the sliding window if it steps down