[SPARK-11623] Sparksql-1.4.1 DataFrameWrite.jdbc() bug

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.1, 1.5.1
• Component/s: Java API, Spark Submit, SQL

Environment

Spark stand alone cluster

• Target Version/s: 1.4.1, 1.5.1
• Assignee: Unassigned
• Reporter: Nguyen Van Nghia
• Created: 10/Nov/15 06:58
• Updated: 24/Nov/15 15:11
• Resolved: 10/Nov/15 08:08

Description

I am running spark-submit in window 8.1 with spark standalone cluster (01 worker and 01 master), the job throw Exception in DataFrameWrite.jdbc(..) scala function.
We found that the following test:
var tableExists = JdbcUtils.tableExists(conn, table)
always return false event if we already created a table.
That drive the function to do creating table from specified DataFrame and the SQL Syntax error for creating table, we locate the SQL execution statement hereafter:
if (!tableExists)

{ val schema = JDBCWriteDetails.schemaString(df, url) val sql = s"CREATE TABLE $table ($schema)" conn.prepareStatement(sql).executeUpdate() // This execution cause sql syntax error }

This happened with spark-1.4.1 and Spark-1.5.1 (our dev environment).
Please help!

Issue Links

• duplicates: SPARK-9078 Use of non-standard LIMIT keyword in JDBC tableExists code
• is cloned by: SPARK-11953 Acknowledge Append Mode in DataFrameWriter