[HUDI-6872] Simplify Out Of Box Schema Evolution Functionality

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: deltastreamer, spark, spark-sql
• Labels: pull-request-available
• Assignee: Jonathan Vexler
• Reporter: Jonathan Vexler
• Created: 18/Sep/23 15:44
• Updated: 25/Jan/24 04:12
• Resolved: 25/Jan/24 04:12

Description

Test schema evolution capabilities out of the box for deltastreamer and datasource. Make schema evolution out of the box easy to understand and use

Issue Links

• blocks: HUDI-6957 Allow column re-ordering for schema on write
• causes: HUDI-7327 hoodie.write.handle.missing.cols.with.lossless.type.promotion does not work with HoodieIncrSource unless meta cols are dropped
• is a parent of: HUDI-6955 Fail the delta streamer pipeline on re-ordering of columns
• relates to: HUDI-6947 Clean up HoodieSparkSqlWriter.deduceWriterSchema
• relates to: HUDI-6958 Update Schema Evolution Documentation
• Testing discovered: HUDI-6873 Clustering MOR applies base files after log files
• Testing discovered: HUDI-6948 HoodieAvroParquetReader sets configs wrong (and prevents schema evolution with non row writer)
• links to: GitHub Pull Request #9743
• links to: GitHub Pull Request #10028
• links to: GitHub Pull Request #10034
• links to: GitHub Pull Request #10041
• links to: GitHub Pull Request #10042