Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
Flink seems to struggle saving data to s3 due to the lack of a truncate method, and in my test this resulted in lots of files with a .valid-length suffix
I’m using a bucketing sink:
return new BucketingSink<Tuple2<String, Object>>(path) .setWriter(writer) .setBucketer(new DateTimeBucketer<Tuple2<String, Object>>(formatString));
See also, the discussion in the comments here: https://issues.apache.org/jira/browse/FLINK-8543?filter=-2
Attachments
Attachments
Issue Links
- duplicates
-
FLINK-9752 Add an S3 RecoverableWriter
- Closed
- is related to
-
FLINK-8543 Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen
- Closed
- relates to
-
FLINK-8543 Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen
- Closed