[HUDI-3304] Support partial update payload

• Type: New Feature
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: writer-core
• Labels: pull-request-available
• Story Points: 2
• Assignee: Jian Feng
• Reporter: Jian Feng

Reviewers

Sagar Sumit, Shiyan Xu

• Created: 24/Jan/22 03:33
• Updated: 13/Jun/24 18:46
• Resolved: 17/Feb/23 23:28

Description

current status

• OverwriteNonDefaultsWithLatestAvroPayload implement partial update behavior in combineAndGetUpdateValue method
• Spark sql also have a 'Merge into' syntax support partial update by ExpressionPayload,
• both OverwriteNonDefaultsWithLatestAvroPayload and ExpressionPayload can not handle partial update in preCombine method, so they can only support partial update with COW table

solution

make preCombine function also support partial update(need pass schema as parameter)

Issue Links

• Blocked: HUDI-4779 Flink Hudi Support Partial Column Update
• incorporates: HUDI-4882 Multiple ordering fields for partial update to handle out-of-order events
• links to: GitHub Pull Request #4676
• links to: GitHub Pull Request #7359
• links to: GitHub Pull Request #9979
• links to: GitHub Pull Request #11435