[FLINK-19425] Correct the usage of BulkWriter#flush and BulkWriter#finish

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Fix Version/s: 1.20.0
• Component/s: Connectors / Common
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: hailong wang
• Created: 26/Sep/20 13:37
• Updated: 11/Mar/24 12:44

Description

From the comments, BulkWriter#finish method should flush all buffer before close.

But some subclasses of it do not flush data. These classes are as follows:

1.AvroBulkWriter#finish

2.HadoopCompressionBulkWriter#finish

3.NoCompressionBulkWriter#finish

4.SequenceFileWriter#finish

We should invoke BulkWriter#flush in this finish methods.

On the other hand, We don't have to  invoke BulkWriter#flush in close method. For BulkWriter#finish will flush all data.

1. HadoopPathBasedPartFileWriter#closeForCommit

2. BulkPartWriter#closeForCommit

3. FileSystemTableSink#OutputFormat#close