[HUDI-8719] File group reader enhancement - Phase 0

• Type: Improvement
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.0.1
• Labels: story
• Assignee: Unassigned
• Reporter: Y Ethan Guo
• Created: 12/Dec/24 00:20
• Updated: 13/Dec/24 00:25

Sub-Tasks

• Simplify precombine and ordering field value handling in the file group reader: Patch Available
• Ensure precombine/ordering fields can only be scalar: Open
• Revisit stats generated in HoodieSparkFileGroupReaderBasedMergeHandle: Open
• Prevent user from setting precombine to field that is not in the table schema: Open
• Allow no precombine field in MOR table: In Progress
• Followup to fix all callers to HoodieLogRecordReader to set the right value for max instant time: Open
• Default to overwrite merge mode if they don't specify an ordering field: Open
• HoodieSparkRecordMerger does not handle deletes based on the preCombine/ordering field: In Progress
• Hoodie FilegroupReader cannot read Enums from MOR avro log blocks: Open
• Delete records are inconsistent depending on MOR/COW, Avro/Spark record merger, new filegroup reader enabled/disabled: Closed
• Revalidate auto-keygen flow with file group reader: Open
• Bridge gaps on delete handling behavior in the file group reader: Open
• Revisit decimal handling for all parquet readers and writers to be consistent: Open
• Revisit confs in Spark broadcast manager set for reader context: Open
• hoodie.datasource.insert.dup.policy interplay with file group reader: Open
• Add bootstrap read testing to TestHoodieFileGroupReaderBase: Open