[SPARK-48030] InternalRowComparableWrapper should cache rowOrdering to improve performace

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1, 3.4.3
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: YE
• Reporter: YE
• Created: 28/Apr/24 12:30
• Updated: 30/Apr/24 06:28
• Resolved: 30/Apr/24 04:26

Description

InternalRowComparableWrapper recreates row ordering for each output partition when SPJ is enabled. The row ordering is generated via codegen which is quite expensive and the output partitions might be quite large for production table such as hundreds of thousands partitions. We encountered this issue when applying SPJ with multiple large Iceberg tables and the plan phase took tens of minutes to complete.

Attaching a screenshot to provide related stack trace:
 

A simple fix for this would be caching the rowOrdering for InternalRowComparableWrapper as the datatype of the InternalRow is immutable

Issue Links

• is related to: SPARK-37375 Umbrella: Storage Partitioned Join (SPJ)
• links to: GitHub Pull Request #46265