[HUDI-6654] Add new log block header type to store record positions

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.14.0, 1.0.0-beta1
• Labels: pull-request-available
• Epic Link: 1.X Format Changes
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Ethan Guo (this is the old account; please use "yihua")
• Created: 05/Aug/23 23:33
• Updated: 15/Aug/23 03:31
• Resolved: 15/Aug/23 03:31

Description

To support position-based merging of base and log files, we need to encode positions in the log blocks so that the positions can be used directly, without having to deserialize records or delete keys for OverwriteWithLatest payload, or with ordering values required only for `DefaultHoodieRecordPayload` supporting event time based streaming.  We add a new `HeaderMetadataType` to store the positions in the log block header.

Issue Links

• is a child of: HUDI-6653 Support position-based merging of base and log files
• links to: GitHub Pull Request #9376