[SPARK-44027] create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: PySpark
• Labels: features newbie
• Assignee: Unassigned
• Reporter: Martin Bode
• Created: 12/Jun/23 17:23
• Updated: 16/Feb/24 12:30

Description

currently only temporary Spark Views can be created from a DataFrame:

• DataFrame.createTempView
• DataFrame.createOrReplaceTempView
• DataFrame.createGlobalTempView
• DataFrame.createOrReplaceGlobalTempView

When a user needs a permanent Spark View he has to fall back to Spark SQL (CREATE VIEW AS SELECT...).

Sometimes it is easier and more readable to specify the desired logic of the view through Scala/PySpark DataFrame API.
Therefore, I'd like to suggest to implement a new PySpark method that allows creating a permanent Spark View from a DataFrame (e.g. DataFrame.createOrReplaceView).

see also:

• https://community.databricks.com/s/question/0D53f00001PANVgCAP/is-there-a-way-to-create-a-nontemporary-spark-view-with-pyspark
• https://lists.apache.org/thread/jzkznvt7cfjhmo77w1tlksxkwyvmvvfb