[HUDI-7665] Rolling upgrade to table version 8

• Type: Task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Labels: hudi-1.0.0-beta2 pull-request-available
• Epic Link: 1.0 Migration
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit

Reviewers

Danny Chen

• Created: 25/Apr/24 06:57
• Updated: Yesterday 21:00

Description

 
Supported migration combinations 

Obviously, within 1.x code line - writers, readers, table services all need to co-exist for both table versions (tv) 6 & 8. 

	Reader 0.x	Reader 1.x tv=6/8	Table Services 0.x	Table Services 1.x tv=6	Table Services 1.x tv=8
Writer 1.x tv=6
Writer 1.x tv=8
Table Services 1.x tv=6
Table Services 1.x tv=8

We are making the practical restriction that table services 0.x, need to be stopped first, upgraded to new binary 1.x, before resuming operations even for tv=6 to be able to co-exist with a 1.x tv=6 writer.

Recommended migration steps 

1. Stop any async table services in 0.x completely. 
2. Upgrade writers to 1.x with tv=6 (this won't auto-upgrade anything); 0.x readers will continue to work; writers can also be readers and will continue to read both tv=6.
3. Upgrade table services to 1.x with tv=6, and resume operations. 
4. Upgrade all remaining readers to 1.x, with tv=6. 
5. Redeploy writers with tv=8; table services and readers will adapt/pick up tv=8 on the fly. 

Things to be done during rolling upgrade 

the 6 to 8 upgrade handler, need to handle the following. 

Migrating table properties including key generators, payloads. Migrating timeline to new layout Full compact the table and clean, to get rid of log files?  Migration path for CDC and incremental queries. Handle differences between 0.x and 1.x for stuff needed in upgrade e.g compaction (need to compact older file slice), rollback (marker differences b/w 0.14 and 0.15)

 

 Safe migration protocol : 

<wip>

 

 Downgrade tooling : 

<wip>

 

 

Issue Links

• links to: GitHub Pull Request #12250
• links to: GitHub Pull Request #12327

Sub-Tasks

• Write archived timeline as LSM timeline: Closed
• Ensure full compaction before downgrade for unsupported features: Closed
• Implement LegacyArchivedMetaEntryWriter to rewrite archived timeline: Closed
• Handle upgrade downgrade with cluster action type for pending clustering instants: Closed
• Change the location of tv=8 timeline to .hoodie/timeline: Closed
• Consider changing MT upgrade as a recursive upgrade of another table: Open
• Handle avro/json serde for commit metadata content: Closed
• Handle log file marker: Closed
• Duplicates after downgrade and clustering: Closed
• Add a check in upgrade path to delete metadata table if behind data table: Closed