[SPARK-39348] Create table in overwrite mode fails when interrupted

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Max
• Created: 31/May/22 13:21
• Updated: 09/Feb/23 12:01

Description

When you attempt to rerun an Apache Spark write operation by cancelling the currently running job, the following error occurs:

Error: org.apache.spark.sql.AnalysisException: Cannot create the managed table('`testdb`.` testtable`').
The associated location ('dbfs:/user/hive/warehouse/testdb.db/metastore_cache_ testtable) already exists.;

This problem can occur if:

• The cluster is terminated while a write operation is in progress.
• A temporary network issue occurs.
• The job is interrupted.

You can reproduce the problem by following these steps:

1. Create a DataFrame:

val df = spark.range(1000)

2. Write the DataFrame to a location in overwrite mode:

df.write.mode(SaveMode.Overwrite).saveAsTable("testdb.testtable")

3. Cancel the command while it is executing.

4. Re-run the write command.