Details
Major Description
`RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception:
IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.
We should switch the order between these two rules to satisfy the requirement when instantiating `PartitioningCollection`.
Attachments
Issue Links
- links to
(2 links to)