[HUDI-3626] Refactor TableSchemaResolver to remove `includeMetadataFields` flags

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Story Points: 4
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 15/Mar/22 00:43
• Updated: 21/Dec/23 15:27

Description

Currently, `TableSchemaResolver` to pass `includeMetadataFields` in its APIs that would selectively remove metadata fields from the returned schemas. There are multiple issues with this flag:

1. It's applied inconsistently: sometimes it just means that meta fields won't be added, and sometimes it means fields would be removed even if present
2. This flag actually spells the usage context into TableSchemaResolver: it's highly contextual whether caller wants to remove, omit such meta-fields, or take the schema as is and should not be spilled into the Resolver itself.
3. B/c of it there's no way to know actually what was actual schema the data was written with (b/c flag might not only omit, but also change the original schema)

Issue Links

• is related to: HUDI-4176 TableSchemaResolver fetches/parses HoodieCommitMetadata multiple times while extracting Schema