Description
Currently, PushProjectionThroughUnion optimizer only supports pushdown project operator to both sides of a Union operator when expression is deterministic , in fact, we can be like pushdown filters, also support pushdown project operator to both sides of a Union operator when expression is non-deterministic , this PR description fix this problem。now the explain looks like:
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion ===
Input LogicalPlan:
Project a#0, rand(10) AS rnd#9
+- Union
:- LocalRelation <empty>, a#0, b#1, c#2
:- LocalRelation <empty>, d#3, e#4, f#5
+- LocalRelation <empty>, g#6, h#7, i#8
Output LogicalPlan:
Project a#0, rand(10) AS rnd#9
+- Union
:- Project a#0
: +- LocalRelation <empty>, a#0, b#1, c#2
:- Project d#3
: +- LocalRelation <empty>, d#3, e#4, f#5
+- Project g#6
+- LocalRelation <empty>, g#6, h#7, i#8