[SPARK-30305] Unable to overwrite table created by another Spark session

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.4
• Component/s: PySpark
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: David Lindelöf
• Created: 19/Dec/19 14:44
• Updated: 25/May/21 01:52
• Resolved: 25/May/21 01:42

Description

I'm unable to save a dataframe to a table, even when passing the `mode='overwrite'` argument:

 

def test_pyspark_can_overwrite_table():
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice')
    spark.stop()
    spark = SparkSession.builder.getOrCreate()
    sdf = spark.createDataFrame([('Alice', 1)])
    sdf.write.saveAsTable('alice', mode='overwrite')

I would expect this to succeed. Instead, I get the output in the attached file. The root exception reads:

pyspark.sql.utils.AnalysisException: "Can not create the managed table('`alice`'). The associated location('file:/Users/dlindelof/Work/lodging-brain-mod-lever/scripts/Python/godfather/spark-warehouse/alice') already exists.;"