Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.18.1
-
None
-
None
-
Our PyFlink SQL jobs are running in AWS EKS environment.
-
Important
Description
Our Flink table SQL job received data from the Kafka streams and then sinked all partitioned data into the associated parquet files under the same S3 folder through the filesystem SQL connector.
For the S3 filesystem SQL connector, sink.partition-commit.policy.kind was set to 'success-file' and sink.partition-commit.trigger was set to 'partition-time'. We found that _SUCCESS file in the S3 folder was generated multiple times after multiple partitions are committed.
Because all partitioned parquet files and _SUCCESS file are in the same S3 folder and _SUCCESS file is used to trigger the downstream application, we really like the _SUCCESS file to be generated only once instead of multiple times after all partitions are committed and all parquet files are ready to be processed. Thus, one _SUCCESS file can be used to trigger the downstream application only once instead of multiple times.
We knew we could set sink.partition-commit.trigger to 'process-time' to generate _SUCCESS file only once in the S3 folder; however, 'process-time' would not meet our business requirements.
We'd request the FileSystem SQL connector should support to the following new user case:
Even if sink.partition-commit.trigger is set to 'partition-time', _SUCCESS file will be generated only once after all partitions are committed and all output files are ready to be processed, and will be used to trigger the downstream application only once instead of multiple times.