[SPARK-16208] Add `PropagateEmptyRelation` optimizer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 25/Jun/16 09:17
• Updated: 10/May/17 00:47
• Resolved: 01/Jul/16 14:15

Description

This issue adds a new logical optimizer, `PropagateEmptyRelation`, to collapse a logical plans consisting of only empty LocalRelations.

*Optimizer Targets*

1. Binary(or Higher)-node Logical Plans

• Union with all empty children.
• Join with one or two empty children (including Intersect/Except).
  2. Unary-node Logical Plans
• Project/Filter/Sample/Join/Limit/Repartition with all empty children.
• Aggregate with all empty children and without AggregateFunction expressions, COUNT.
• Generate with Explode because other UserDefinedGenerators like Hive UDTF returns results.

*Sample Query*

WITH t1 AS (SELECT a FROM VALUES 1 t(a)),
     t2 AS (SELECT b FROM VALUES 1 t(b) WHERE 1=2)
SELECT a,b
FROM t1, t2
WHERE a=b
GROUP BY a,b
HAVING a>1
ORDER BY a,b

*Before*

scala> sql("with t1 as (select a from values 1 t(a)), t2 as (select b from values 1 t(b) where 1=2) select a,b from t1, t2 where a=b group by a,b having a>1 order by a,b").explain
== Physical Plan ==
*Sort [a#0 ASC, b#1 ASC], true, 0
+- Exchange rangepartitioning(a#0 ASC, b#1 ASC, 200)
   +- *HashAggregate(keys=[a#0, b#1], functions=[])
      +- Exchange hashpartitioning(a#0, b#1, 200)
         +- *HashAggregate(keys=[a#0, b#1], functions=[])
            +- *BroadcastHashJoin [a#0], [b#1], Inner, BuildRight
               :- *Filter (isnotnull(a#0) && (a#0 > 1))
               :  +- LocalTableScan [a#0]
               +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
                  +- *Filter (isnotnull(b#1) && (b#1 > 1))
                     +- LocalTableScan <empty>, [b#1]

*After*

scala> sql("with t1 as (select a from values 1 t(a)), t2 as (select b from values 1 t(b) where 1=2) select a,b from t1, t2 where a=b group by a,b having a>1 order by a,b").explain
== Physical Plan ==
LocalTableScan <empty>, [a#0, b#1]

Issue Links

• is related to: SPARK-20686 PropagateEmptyRelation incorrectly handles aggregate without grouping expressions
• links to: [Github] Pull Request #13906 (dongjoon-hyun)