[FLINK-33059] Support transparent compression for file-connector for all file input formats

• Type: Technical Debt
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.19.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Etienne Chauchot
• Reporter: Etienne Chauchot
• Created: 07/Sep/23 10:09
• Updated: 10/Nov/23 08:48
• Resolved: 10/Nov/23 08:48

Description

Some FileInputFormats don't use FileInputFormat#createSplits (that would detect that the file is non-splittable and deal with reading boundaries correctly), they all create split manually from FileSourceSplit. If input files are compressed, split length is determined by the compressed file length leading to this bug. We should force reading the whole file split (like it is done for binary input formats) on compressed files. Parallelism is still done at the file level (as now)

Issue Links

• links to: GitHub Pull Request #23443