[FLINK-8939] Provide better support for saving streaming data to s3

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: API / DataStream, Connectors / Common
• Assignee: Unassigned
• Reporter: chris snow
• Created: 14/Mar/18 07:51
• Updated: 24/Sep/18 09:27
• Resolved: 24/Sep/18 09:27

Description

Flink seems to struggle saving data to s3 due to the lack of a truncate method, and in my test this resulted in lots of files with a .valid-length suffix

I’m using a bucketing sink:

return new BucketingSink<Tuple2<String, Object>>(path)
.setWriter(writer)
.setBucketer(new DateTimeBucketer<Tuple2<String, Object>>(formatString));

 

See also, the discussion in the comments here: https://issues.apache.org/jira/browse/FLINK-8543?filter=-2

Issue Links

• duplicates: FLINK-9752 Add an S3 RecoverableWriter
• is related to: FLINK-8543 Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen
• relates to: FLINK-8543 Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen