[SPARK-26764] [SPIP] Spark Relational Cache

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Adrian Wang
• Created: 29/Jan/19 09:00
• Updated: 29/Jun/21 02:26

Description

In modern database systems, relational cache is a common technology to boost ad-hoc queries. While Spark provides cache natively, Spark SQL should be able to utilize the relationship between relations to boost all possible queries. In this SPIP, we will make Spark be able to utilize all defined cached relations if possible, without explicit substitution in user query, as well as keep some user defined cache available in different sessions. Materialized views in many database systems provide similar function.

Issue Links

• relates to: SPARK-29038 SPIP: Support Spark Materialized View