[HUDI-7237] Minor Improvements to Schema Handling in Delta Sync

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.15.0, 1.0.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Timothy Brown
• Created: 17/Dec/23 01:35
• Updated: 07/Jun/24 01:11
• Resolved: 07/Jun/24 01:11

Description

There are a two minor items that we have run into running DeltaStreamer in production.
1. The number of times the schema is fetched is more than it needs to be and can put unnecessary load on schema providers or increase file system reads

2. SchemaProviders that return null target schemas on empty batches cause null schema values in commits leading to unexpected issues later

 

Issue Links

• relates to: HUDI-7417 NPE in StreamSync if empty batch and schema and upsert operation new table
• links to: GitHub Pull Request #10342