[FLINK-19162] Allow Split Reader based sources to reuse record batches

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: Connectors / Common
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 07/Sep/20 18:03
• Updated: 15/Nov/20 22:06
• Resolved: 11/Sep/20 16:37

Description

The Split Readers hand over a batch of records at a time from the I/O thread (fetching and decoding) to the main operator processing thread.

 These structures can memory intensive and expensive and performance greatly benefits from reusing them. This is especially true for high-performance format readers like ORC and Parquet.

While previous sources (where I/O was in the main thread) could reuse objects in a trivial manner, the new Split Reader API (with multiple threads) needs an explicit recycle() hook to allow returning/reusing these objects.