[SPARK-15712] Proper temp table support

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 01/Jun/16 19:05
• Updated: 21/May/19 04:35
• Resolved: 21/May/19 04:35

Description

For proper temp table support, I am proposing to create a temp dir for every SparkSession. This can be done through the following changes:

1. We add a root temp dir (the path is configurable).
2. For every SparkSession, we create a temp dir for this SparkSession in the root temp dir.

To make the semantic of CREATE TEMP TABLE ... USING ... OPTIONS ... AS SELECT easier to understand, I am proposing the following changes:

1. We only allow FileFormat data sources used by this command.
2. We make USING clause optional and by default we use the data source set in spark.sql.sources.default.
3. We do not allow users to add path in the OPTIONS
4. We save the temp table in the temp location associated with the current SparkSession.

Issue Links

• relates to: SPARK-15711 Ban CREATE TEMP TABLE USING AS SELECT for now