[HUDI-957] Umbrella ticket for sequencing common tasks required to progress/unblock RFC-08, RFC-15 & RFC-19

• Type: Epic
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: Common Core, compaction, index, storage-management
• Assignee: Nishith Agarwal
• Reporter: Nishith Agarwal
• Created: 26/May/20 19:00
• Updated: 09/May/22 09:10
• Resolved: 23/Sep/20 02:30

Description

There are 3 different designs proposed in following RFC's 08, 15 & 19. On further analysis there are a bunch of common changes that will benefit all 3 and some that are specific to each individual design. This ticket is to track most of the common changes so those can be parallelized and all members in the community can help contribute to land these soon.

Issue Links

• is a child of: HUDI-842 RFC-15 : Implementation of File Listing elimination
• is a parent of: HUDI-435 Make async compaction/cleaning extensible to new usages
• is a parent of: HUDI-684 Introduce abstraction for writing and reading and compacting from FileGroups

Sub-Tasks

• Implementation for managing compaction across multiple Hudi tables (index, metadata, data..): Open
• HoodieTable abstraction with pluggable Base and Log formats: Resolved
• HFile Support for HUDI: Closed
• Implement HoodieHFileDataBlock to allow for logging Hfile to log blocks: Closed
• HoodieLogFormatWriter and HoodieLogFileReader support for HoodieHFileDataBlock: Closed
• HoodieMergedLogRecordScanner support for HoodieHFileDataBlocks: Closed
• Multi Table Commits when modeling Index, consolidated metadata and data as a Hoodie Table: Open
• HUDICLI support for new table abstraction implementation: Open
• Rehaul hoodie table tests to absorb the hoodie table abstraction design: Open
• Configs rehaul to support table abstraction for metadata table, index table etc: Closed