[HUDI-774] Spark to Avro converter incorrectly generates optional fields

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Labels: pull-request-available sev:critical user-support-issues
• Assignee: Unassigned
• Reporter: Alexander Filipchik
• Created: 09/Apr/20 02:05
• Updated: 03/Jun/22 16:15
• Resolved: 23/Apr/21 20:25

Description

I think https://issues.apache.org/jira/browse/SPARK-28008 is a good descriptions of what is happening.

 

It can cause a situation when schema in the MOR log files is incompatible with the schema produced by RowBasedSchemaProvider, so compactions will stall.

 

I have a fix which is a bit hacky -> postprocess schema produced by the converter and

1) Make sure unions with null types have those null types at position 0

2) They have default values set to null

I couldn't find a way to do a clean fix as some classes that are problematic are from Hive and called from Spark.

Issue Links

• links to: GitHub Pull Request #1514
• links to: GitHub Pull Request #1514