[FLINK-32213] Add get off heap buffer in memory segment

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Runtime / Task
• Labels: pull-request-available
• Assignee: Fang Yong
• Reporter: Fang Yong
• Created: 30/May/23 01:55
• Updated: 12/Jun/23 17:18
• Resolved: 12/Jun/23 17:18

Description

When flink job writes data to data lake such as paimon, iceberg and hudi, the sink will write data to writer buffer first, then flush the data to file system. To manage the writer buffer better, we'd like to allocate segment from managed memory in flink and get off heap buffer to create writer buffer

Issue Links

• links to: GitHub Pull Request #22675