[FLINK-3637] Change RollingSink Writer interface to allow wider range of outputs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Connectors / Common
• Labels: features
• Assignee: Lasse Dalegaard
• Reporter: Lasse Dalegaard
• Created: 18/Mar/16 13:45
• Updated: 11/Jan/17 12:50
• Resolved: 06/Apr/16 09:33

Description

Currently the RollingSink Writer interface only works with FSDataOutputStreams, which precludes it from being used with some existing libraries like Apache ORC and Parquet.

To fix this, a new Writer interface can be created, which receives FileSystem and Path objects, instead of FSDataOutputStream.

To ensure exactly-once semantics, the Writer interface must also be extended so that the current write-offset can be retrieved at checkpointing time. For formats like ORC this requires a footer to be written, before the offset is returned. Checkpointing already calls flush on the writer, but either flush needs to return the current length of the output file, or alternatively a new method has to be added for this.

The existing Writer interface can be recreated with a wrapper on top of the new Writer interface. The existing code that manages the FSDataOutputStream can then be moved into this new wrapper.

Issue Links

• links to: GitHub Pull Request #1826