[FLINK-26450] FileStateHandle.discardState does not process return value

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.13.6, 1.14.3, 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Connectors / FileSystem, Runtime / Coordination
• Labels: pull-request-available
• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 02/Mar/22 13:15
• Updated: 10/Mar/22 12:43
• Resolved: 10/Mar/22 12:07

Description

The retryable cleanup does not work properly if there's an error appearing during the FileSystem.delete call which is used within FileStateHandle.discardState. Some FileSystem implementations (e.g. S3 presto; see PrestoS3FileSystem:512 through PrestoS3FileSystem.delete(Path, boolean)) return false in case of an error which will be swallowed in FileStateHandle.discardState.

Issue Links

• Discovered while testing: FLINK-26388 Release Testing: Repeatable Cleanup (FLINK-25433)
• is caused by: FLINK-4910 Introduce safety net for closing file system streams
• is fixed by: FLINK-25484 TableRollingPolicy do not support inactivityInterval config which is supported in datastream api
• relates to: FLINK-26488 FileSystem.listFiles is not implemented consistently
• relates to: FLINK-26484 FileSystem.delete is not implemented consistently
• links to: GitHub Pull Request #18963