[HUDI-4176] TableSchemaResolver fetches/parses HoodieCommitMetadata multiple times while extracting Schema

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.11.1
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Performance Improvements
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 01/Jun/22 21:30
• Updated: 07/Jun/22 06:53
• Resolved: 07/Jun/22 06:53

Description

We've recently discovered that TableSchemaResolver does a lot of throw-away work during initialization and basic schema reading performed by Spark Datasource (see screenshot).

This poses a problem for large tables where HoodieCommitMetadata is of non-trivial size (100s of Mbs).

We'd minimize amount of throw-away work done by `TableSchemaResolver` and try to re-use read/parsed commits' metadata as much as possible.

Issue Links

• relates to: HUDI-3626 Refactor TableSchemaResolver to remove `includeMetadataFields` flags
• relates to: HUDI-4178 Performance regressions in Spark DataSourceV2 Integration
• links to: GitHub Pull Request #5733