[HUDI-603] HoodieDeltaStreamer should periodically fetch table schema update

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: deltastreamer
• Labels: evolution pull-request-available schema
• Assignee: Pratyaksh Sharma
• Reporter: Yixue Zhu
• Created: 06/Feb/20 18:25
• Updated: 21/Jan/21 05:40
• Resolved: 21/Jan/21 05:40

Description

HoodieDeltaStreamer create SchemaProvider instance and delegate to DeltaSync for periodical sync. However, default implementation of SchemaProvider does not refresh schema, which can change due to schema evolution. DeltaSync snapshot the schema when it creates writeClient, using the SchemaProvider instance or pick up from source, and the schema for writeClient is not refreshed during the loop of Sync.

I think this needs to be addressed to support schema evolution fully.

Issue Links

• links to: GitHub Pull Request #1566
• links to: GitHub Pull Request #1566