[SPARK-16576] Move plan SQL generation code from SQLBuilder into logical operators

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0
• Component/s: SQL
• Assignee: Unassigned Assign to me
• Reporter: Reynold Xin
• Created: 15/Jul/16 20:51
• Updated: 01/Nov/16 21:54
• Resolved: 01/Nov/16 21:53

Description

We currently handle all SQL query generation in a single class (SQLBuilder). This has many disadvantages:

1. It is not extensible, i.e. it is not possible to introduce a new logical operator, even just for experimentation purpose, without modifying Spark.

2. It is very fragile. When we introduce a new logical operator, it is very likely that we forget to update SQLBuilder and then the use of that new logical operator would fail view definition.

We should move the SQL definition part into logical operators themselves, so this becomes more robust and scalable.

Issue Links

• is related to: SPARK-11012 Canonicalize view definitions
• relates to: SPARK-16590 Improve LogicalPlanToSQLSuite to check generated SQL directly
• relates to: SPARK-18209 More robust view canonicalization without full SQL expansion

Sub-Tasks

• Add subquery SQL building tests to LogicalPlanToSQLSuite: Resolved
• Improve LogicalPlanToSQLSuite to check generated SQL directly: Resolved
• Generate stable SQLs in SQLBuilder: Resolved
• Add `sql` function to LogicalPlan and `NonSQLPlan` trait: Closed