[HUDI-7229] Enable partial updates for CDC work payload

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Labels: pull-request-available
• Epic Link: 1.X Format Changes
• Assignee: Vinoth Chandar
• Reporter: Lin Liu
• Created: 13/Dec/23 23:04
• Updated: 03/May/24 02:13

Description

OLTP workloads on upstream databases, often update/delete/insert different columns in the table on each operation. Currently, Hudi can only supporting partial updates in cases where the same columns are being mutated in a given write to Hudi (e.g Spark SQL ETLs with MIT or Update statements). Here, we explore what it takes to support a smarter storage format, that can only encode the changed columns into log along with the different implementations.

Goals

1. Enable partial update functionality for all existing and potential future CDC workloads without huge modification or duplication.
2. Performance parity with current full-record updates or partial updates across the same set of columns
3. Exhibit reduction in storage costs, by only storing the changed columns.
4. Should also result in computation cost reductions by scanning/processing less data
5. Should not affect the scalability of the existing system ingestion system. The number of files generated for partial update should not increase dramatically.

 

Issue Links

• links to: GitHub Pull Request #10384

Sub-Tasks

• Create one-pager for the design and status of partial updates for CDC: Closed
• Improve the logic of AppendHandle for partial updates: In Progress
• Add proper unit tests and function tests for partial updates: Open
• Test partial updates with delta streamer: Open
• Make code in delta streamer to prepare the data and schema: Open
• Roll out partial update to production: Open