[FLINK-36526] Optimize the overhead of writing with direct buffer in ForSt

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Zakelly Lan
• Reporter: Zakelly Lan
• Created: 14/Oct/24 07:53
• Updated: 15/Oct/24 10:52
• Resolved: 15/Oct/24 10:52

Description

Currently, the ForSt gives a direct buffer to ByteBufferWritableFSDataOutputStream, where the data will be written one byte by byte. According our perf, the statistics of hadoop based fs will be updated once for each byte, which takes a lot of CPU. Below is a flamegraph, where the statistics part is marked as purple (taking 8.14% of the overall CPU).

 

It might be better to copy to a heap buffer before invoking write.

Issue Links

• links to: GitHub Pull Request #25508