[FLINK-20221] DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.2, 1.11.2, 1.12.0
• Fix Version/s: 1.11.3, 1.12.1, 1.13.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Gyula Fora
• Reporter: Gyula Fora
• Created: 18/Nov/20 13:32
• Updated: 08/Dec/20 11:36
• Resolved: 08/Dec/20 11:36

Description

It seems that the delimited input format cannot correctly restore input splits if they belong to compressed files. Basically when a compressed filesplit is restored in the middle, it won't read it anymore leading to dataloss.

The cause of the problem is that for compressed splits that use an inflater stream, the splitlength is set to the magic number -1 which is ignored in the reopen method and causes the split to go to `end` state immediately.

The problem and the fix is shown in this commit:
https://github.com/gyfora/flink/commit/4adc8ba8d1989fff2db43881c9cb3799848c6e0d

Issue Links

• links to: GitHub Pull Request #14174
• links to: GitHub Pull Request #14288
• links to: GitHub Pull Request #14289