[SPARK-17081] Empty strings not preserved which causes SQLException: mismatching column value count

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.4.1
• Fix Version/s: 2.0.0
• Component/s: SQL
• Labels: dataframe empty jdbc null sql
• Flags: Important
• Assignee: Unassigned
• Reporter: Ian Hellstrom
• Created: 16/Aug/16 07:26
• Updated: 12/Dec/22 18:10
• Resolved: 10/Jan/17 12:56

Description

When writing a DataFrame that contains empty strings as values to an RDBMS, the query that is generated does not have the correct column count:

CREATE TABLE demo(foo INTEGER, bar VARCHAR(10));
-----
case class Record(foo: Int, bar: String)
val data = sc.parallelize(List(Record(1, ""))).toDF
data.write.mode("append").jdbc(...)

This causes:

java.sql.SQLException: Column count doesn't match value count at row 1

Proposal: leave empty strings as they are or convert these to NULL (although that may not be what's intended by the user, so make this configurable).