[SPARK-3212] Improve the clarity of caching semantics

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SQL
• Target Version/s: 1.2.0
• Assignee: Michael Armbrust
• Reporter: Michael Armbrust
• Created: 25/Aug/14 19:48
• Updated: 03/Oct/14 19:34
• Resolved: 03/Oct/14 19:34

Description

Right now there are a bunch of different ways to cache tables in Spark SQL. For example:

• tweets.cache()
• sql("SELECT * FROM tweets").cache()
• table("tweets").cache()
• tweets.cache().registerTempTable(tweets)
• sql("CACHE TABLE tweets")
• cacheTable("tweets")

Each of the above commands has subtly different semantics, leading to a very confusing user experience. Ideally, we would stop doing caching based on simple tables names and instead have a phase of optimization that does intelligent matching of query plans with available cached data.

Issue Links

• contains: SPARK-3641 Correctly populate SparkPlan.currentContext
• is related to: SPARK-2189 Method for removing temp tables created by registerAsTable
• is related to: SPARK-3298 [SQL] registerAsTable / registerTempTable overwrites old tables
• relates to: SPARK-1379 Calling .cache() on a SchemaRDD should do something more efficient than caching the individual row objects.
• links to: [Github] Pull Request #2501 (marmbrus)