[PARQUET-2106] BinaryComparator should avoid doing ByteBuffer.wrap in the hot-path

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.2
• Fix Version/s: 1.12.3
• Component/s: parquet-mr
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 03/Dec/21 23:28
• Updated: 19/May/22 06:06
• Resolved: 09/Dec/21 09:31

Description

Background

While writing out large Parquet tables using Spark, we've noticed that BinaryComparator is the source of substantial churn of extremely short-lived `HeapByteBuffer` objects – It's taking up to 16% of total amount of allocations in our benchmarks, putting substantial pressure on a Garbage Collector:

profile_48449_alloc_1638494450_sort_by.html

 

Proposal

We're proposing to adjust lexicographical comparison (at least) to avoid doing any allocations, since this code lies on the hot-path of every Parquet write, therefore causing substantial churn amplification.

 

 

 

Issue Links

• is depended upon by: PARQUET-2145 Release 1.12.3
• relates to: HUDI-2948 Hudi Clustering Performance