[SPARK-28303] Support DELETE/UPDATE/MERGE Operations in DataSource V2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Xianyin Xin Assign to me
• Reporter: Xianyin Xin
• Created: 08/Jul/19 12:10
• Updated: 01/Dec/22 06:28
• Resolved: 07/May/20 06:52

Description

Now many datasources (delta, jdbc, hive with transaction support, kudu, etc) supports deleting/updating data. It's necessary to add related APIs in the datasource V2 API sets.

For example, we suggest add the below interface in V2 API,

SupportsDelete.java

public interface SupportsDelete {
  WriteBuilder delete(Filter[] filters); 
}

which can delete data by simple predicates (complicated cases like correlated subquery is not considered currently).

 

Issue Links

• is duplicated by: SPARK-28909 ANSI SQL: delete and update does not support in Spark

Sub-Tasks

• Support DELETE in DataSource V2: Resolved
• support UPDATE in the parser and add the corresponding logical plan: Resolved
• support MERGE INTO in the parser and add the corresponding logical plan: Resolved
• [SPARK-28351] FollowingUp: Change whereClause to be optional in DELETE: Resolved
• Move DELETE/UPDATE/MERGE relative rules to dmlStatementNoWith to support cte.: Resolved
• support MERGE INTO in the parser and add the corresponding logical plan: Resolved
• Remove the unnecessary conversion from Statement to LogicalPlan for DELETE/UPDATE: Resolved
• MergeIntoTable should avoid ambiguous reference: Resolved