[FLINK-24459] Performance improvement of file sink on Nexmark

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Alexander Trushev
• Reporter: Alexander Trushev
• Created: 06/Oct/21 07:43
• Updated: 15/Oct/21 12:33
• Resolved: 11/Oct/21 04:13

Description

Context

PartitionPathUtils.escapePathName is a pretty simple method that takes String, allocates StringBuilder, appends original or escaped chars, and outputs the result String.

Filesystem sink calls the method several times for each element to determine bucket id. Because of this, it is a hot spot on a workload that writes intensively to filesystem, such as nexmark q10. On my local machine escaping of chars takes 9.53% CPU and 17.8% mem allocations of the whole TaskManager process.

Proposal

PartitionPathUtils.escapePathName improvements

1. Use more efficient Integer.toHexString instead of String.format
2. Do not allocate new string when there is no escapable char in the original string
3. Allocate StringBuilder depending on the original string length instead of the default value

Benefit

Experiment on local machine.
1 TaskManager with 6 slots. Job parallelism 6. Nexmark default configuration + object reuse option.
Before: flink-1.14.0
After: flink-1.14.0 + patch with the improvements

Nexmark q10	Before	After
CPU samples of escapePathName() (% of all)	9.53	1.64
Memory allocations by escapePathName() (% of all)	17.8	2.98
Throughput/Cores (K/s)	107.64	119.42

Diff: CPU -7.89%, Memory -14.82%, Throughput +10.9%

Profiling reports are in the attachment.

Issue Links

• links to: GitHub Pull Request #17416