Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.3.0
-
None
Description
We still find HashClusteredDistribution be useful for batch query as well. For example, we had a case with lower parallelism than expected due to the fact ClusteredDistribution is used for aggregation which matches with HashPartitioning with sub-key groups (note that the technical parallelism also depends on "cardinality" - picking sub-key groups means having less cardinality).
We propose to introduce a new config to require all cluster keys on Aggregate, leveraging HashClusteredDistribution. That said, we propose to rename back HashClusteredDistribution with retaining NOTE for stateful operator. The distribution should not be still touched anyway due to the requirement of stateful operator, but can be co-used with batch case if needed.
