[SPARK-44262] JdbcUtils hardcodes some SQL statements

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Jia Fan
• Reporter: Florent BIVILLE
• Created: 30/Jun/23 09:09
• Updated: 12/Mar/24 14:43
• Resolved: 16/Oct/23 08:56

Description

I am currently investigating an integration with the Neo4j JBDC driver and a Spark-based cloud vendor SDK.

 

This SDK relies on Spark's JdbcUtils to run queries and insert data.

While JdbcUtils partly delegates to org.apache.spark.sql.jdbc.JdbcDialect for some queries, some others are hardcoded to SQL, see:

• org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#dropTable
• org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#getInsertStatement

 

This works fine for relational databases but breaks for NOSQL stores that do not support SQL translation (like Neo4j).

Is there a plan to augment the JdbcDialect surface so that it is also responsible for these currently-hardcoded queries?

Issue Links

• links to: GitHub Pull Request #41855