[GOBBLIN-1345] Implement a Kafka schema change detecting converter

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-kafka
• Assignee: Shirshanka Das
• Reporter: Sudarshan Vasudevan
• Created: 18/Dec/20 22:27
• Updated: 14/Jun/23 04:13

Description

For long-running Gobblin pipelines such as Gobblin-Kafka with streaming mode enabled, we need the ability for the pipeline to detect upstream schema changes to enable proper handling by downstream constructs. For instance, in the case of sinks such as HDFS and blob stores, such schema changes should result in the writer closing any open files.