[FLINK-24728] Batch SQL file sink forgets to close the output stream

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.4, 1.14.0, 1.12.5, 1.13.3
• Fix Version/s: 1.13.6, 1.14.3, 1.15.0
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Caizhi Weng
• Reporter: Caizhi Weng
• Created: 02/Nov/21 07:58
• Updated: 17/Dec/21 05:39
• Resolved: 17/Dec/21 05:39

Description

I tried to write a large avro file into HDFS and discover that the displayed file size in HDFS is extremely small, but copying that file to local yields the correct size. If we create another Flink job and read that avro file from HDFS, the job will finish without outputting any record because the file size Flink gets from HDFS is the very small file size.

This is because the output format created in FileSystemTableSink#createBulkWriterOutputFormat only finishes the BulkWriter. According to the java doc of BulkWriter#finish bulk writers should not close the output stream and should leave them to the framework.

Issue Links

• links to: GitHub Pull Request #17638
• links to: GitHub Pull Request #17655
• links to: GitHub Pull Request #18073
• links to: GitHub Pull Request #18074