[SPARK-28612] DataSourceV2: Add new DataFrameWriter API for v2

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 03/Aug/19 23:17
• Updated: 12/Dec/22 18:11
• Resolved: 19/Sep/19 20:33

Description

This tracks adding an API like the one proposed in SPARK-23521:

df.writeTo("catalog.db.table").append() // AppendData
df.writeTo("catalog.db.table").overwriteDynamic() // OverwritePartiitonsDynamic
df.writeTo("catalog.db.table").overwrite($"date" === '2019-01-01') // OverwriteByExpression
df.writeTo("catalog.db.table").partitionBy($"type", $"date").create() // CTAS
df.writeTo("catalog.db.table").replace() // RTAS

Issue Links

• blocks: SPARK-29157 DataSourceV2: Add DataFrameWriterV2 to Python API
• links to: [Github] Pull Request #29568 (HeartSaVioR)
• links to: [Github] Pull Request #29568 (HeartSaVioR)
• links to: GitHub Pull Request #25354
• links to: GitHub Pull Request #25681