[SPARK-47612] Improve picking the side of partially clustered distribution accroding to partition size

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Qi Zhu
• Created: 27/Mar/24 05:54
• Updated: 27/Mar/24 06:05

Description

Now we pick up the side of partially clustered distribution:

SPJ currently relies on a simple heuristic and always pick the side with less data size based on table statistics as the side fully clustered, even though it could also contain skewed partitions.

We can potentially do fine-grained comparison based on partition values, since we have the information now.