[HUDI-5633] Fixing HoodieSparkRecord performance bottlenecks

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Labels: pull-request-available
• Story Points: 4
• Epic Link: RFC-46: Engine Native Record Payloads
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 28/Jan/23 03:32
• Updated: 23/Feb/23 03:07
• Resolved: 01/Feb/23 22:26

Description

There currently following issues w/ the current HoodieSparkRecord implementation:

1. It rewrites records using `rewriteRecord` and `rewriteRecordWithNewSchema` which do Schema traversals for every record. Instead we should do schema traversal only once and produce a transformer that will directly create new record from the old one.
2. Records are currently copied for every Executor even for Simple one which actually is not buffering any records and therefore doesn't require records to be copied.

Issue Links

• is duplicated by: HUDI-5641 Streamline Advanced Schema Evolution flow
• links to: GitHub Pull Request #7769