[HUDI-2706] Refactor Spark-SQL to make consistent with DataFrame

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 0.10.0
• Component/s: spark
• Labels: pull-request-available
• Assignee: Yann Byron
• Reporter: Yann Byron

Reviewers

Shaofeng Li, Shiyan Xu

• Created: 07/Nov/21 02:04
• Updated: 06/Mar/24 02:24

Description

1. refactor `CreateHoodieTableCommand`;
   1. use `TBLPROPERTIES` syntax to pass config rather than `OPTIONS`, and keep compatible with `OPTIONS` syntax;
   2. force to provide `PrimaryKey`, so that make Update/Delete available;
2. hudi spark-sql decouple from metastore as far as possible, get config from local hoodie.properties first;
3. modify operation when insert/merge, make consistent with dataframe api.
4. add parameter validation for recordKey, preCombineKey, keyGenerator, even if those parameters are defined by different keys.

Issue Links

• fixes: HUDI-2505 [UMBRELLA] Spark DataSource APIs and Spark SQL discrepancies
• fixes: HUDI-2234 MERGE INTO works only ON primary key
• links to: GitHub Pull Request #3936