[SPARK-11953] Acknowledge Append Mode in DataFrameWriter

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.4.1, 1.5.1
• Component/s: Java API, Spark Submit, SQL

Environment

Spark stand alone cluster

• Assignee: Unassigned Assign to me
• Reporter: Siva Gudavalli
• Created: 24/Nov/15 15:11
• Updated: 25/Nov/15 21:52
• Resolved: 25/Nov/15 21:52

Description

DataFramewriter is not acknowledging Append Mode.

In case of Append Mode => It is verfiying if the table exists and creating a new table if it is not there and then Inserting data.
If the table exists check is failed for whatsover reason, it still assumes table doesnt exists and creates a new table.

As per my understanding, Append should perform "no DDL". It is okay to throw error saying "table doesnt exist for Inserts'. But creating a Table should not be allowed in Append Mode.

I believe we should do two things here

perform DDL => if and only if table exists check results "Not exists"
do not perform DDL => if SaveMode is Append

Issue Links

• is a clone of: SPARK-11623 Sparksql-1.4.1 DataFrameWrite.jdbc() bug