[FLINK-11794] Allow compression of row format files created by StreamingFileSink

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.7.2
• Component/s: Connectors / FileSystem
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Elias Levy
• Created: 02/Mar/19 07:28
• Updated: 28/Nov/21 22:38

Description

Currently, there is no mechanism to compress files created using a StreamingFileSink.  This is highly desirable when output is a text based row format such as JSON.

Possible alternatives are the introduction of a callback that gets passed the local file before it is uploaded to the DFS, so that it could be compressed; or a factory method could be used that returns an OutputStream, such as GZIPOutputStream, that compresses a passed in output stream that could be then used by the Encoder.