[FLINK-21642] RequestReplyFunction recovery fails with a remote SDK

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: statefun-3.0.0
• Component/s: Stateful Functions
• Labels: pull-request-available
• Assignee: Igal Shilman
• Reporter: Igal Shilman
• Created: 06/Mar/21 20:28
• Updated: 10/Mar/21 06:46
• Resolved: 10/Mar/21 06:46

Description

While extending our smoke e2e test to use the remote SDKS I've stumbled upon a bug in the RequestReplyFunction. We get a unknown state exception after recovery.

The exact scenario that trigger that bug is:

1. There was  request in flight.
2. A  failure occurs that causes the job to restart.
3. On restore, we start with no managed state
4. But we try to re-send to the SDK exactly the same ToFunction message.
5. That ToFunction contains state definitions from the previous attempt. (before the failure)
6. The SDK processes this message normally (it has all the state definitions that it knows)
7. The SDK responds with a state mutation.
8. The PersistedRemoteFunctionValues fails with unknown state. 

 

We need to treat the ToFunction messages as a retryBatch, instead of sending it as-is.

 

Issue Links

• links to: GitHub Pull Request #208