[SPARK-50228] Move the RewriteCollationJoin rule to FinishAnalysis

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 05/Nov/24 16:51
• Updated: 1 week ago 07:57
• Resolved: 1 week ago 07:57

Description

Such conversions should be NOT considered as an optimization, but it is needed for correctness. So, we cannot run the substitutions as an optimization rule, it should be applied before any.

We could modify RewriteCollationJoin and take into account DPP, but we might miss something else. Fxing ALL results of optimizations in RewriteCollationJoin introduces some dependencies.

Issue Links

• is a clone of: SPARK-48000 Hash join support for strings with collation (StringType only)
• links to: GitHub Pull Request #48759