[FLINK-5944] Flink should support reading Snappy Files

• Type: New Feature
• Status: In Progress
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Connectors / Hadoop Compatibility
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned features
• Assignee: Unassigned
• Reporter: Ilya Ganelin
• Created: 01/Mar/17 18:41
• Updated: 05/Jan/22 22:38

Description

Snappy is an extremely performant compression format that's widely used offering fast decompression/compression.

This can be easily implemented by creating a SnappyInflaterInputStreamFactory and updating the initDefaultInflateInputStreamFactories in FileInputFormat.

Flink already includes the Snappy dependency in the project.

There is a minor gotcha in this. If we wish to use this with Hadoop, then we must provide two separate implementations since Hadoop uses a different version of the snappy format than Snappy Java (which is the xerial/snappy included in Flink).

Issue Links

• links to: GitHub Pull Request #4683