Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
3.0.0
-
None
-
None
Description
Split an expand into several small Expand, which contains the Specified number of projections.
For instance, like this sql.select a, b, c, d, count(1) from table1 group by a, b, c, d with cube. It can expand 2^4 times of original data size.
Now we specify the spark.sql.optimizer.projections.size=4.The Expand will be split into 2^4/4 smallExpand.It can reduce the shuffle pressure and improve performance in multidimensional analysis when data is huge.
