[FLINK-26284] The ZooKeeperStateHandleStore cleans the metadata before cleaning the StateHandle

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 21/Feb/22 14:27
• Updated: 01/Apr/22 12:59
• Resolved: 28/Feb/22 10:35

Description

Cleanup of job state does not work properly in an HA setup. releaseAndTryRemove deletes the meta data stored in the store before cleaning up the StateHandle. If the StateHandle cleanup fails after the reference is already deleted in the StateHandleStore, a cleanup retry will constantly fail because it cannot deserialize the StateHandle anymore.

Issue Links

• causes: FLINK-26987 ZooKeeperStateHandleStore.getAllAndLock ends up in a infinite loop if there's an entry marked for deletion that's not cleaned up, yet
• is blocked by: FLINK-26285 ZooKeeperStateHandleStore does not handle not existing nodes properly in getAllAndLock
• is caused by: FLINK-25432 Introduce common interfaces for cleaning up local and global job data
• is cloned by: FLINK-26286 The KubernetesStateHandleStore cleans the metadata before cleaning the StateHandle
• Testing discovered: FLINK-26285 ZooKeeperStateHandleStore does not handle not existing nodes properly in getAllAndLock
• Testing discovered: FLINK-26288 Investigate why RetrievableStreamStateHandle.close does not call the close method of the wrapped instance
• links to: GitHub Pull Request #18869