[FLINK-19121] Avoid accessing HDFS frequently in HiveBulkWriterFactory

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.11.1, 1.12.0
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 02/Sep/20 05:56
• Updated: 19/Feb/21 07:28
• Resolved: 18/Sep/20 06:29

Description

In HadoopPathBasedBulkWriter, getSize will invoke `FileSystem.exists` and `FileSystem.getFileStatus`, but it is invoked per record.

There will be lots of visits to HDFS, may make HDFS pressure too high.

Issue Links

• links to: GitHub Pull Request #13301