[HUDI-863] nested structs containing decimal types lead to null pointer exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.3
• Labels: bug-bash-0.6.0 pull-request-available
• Assignee: Roland Johann
• Reporter: Roland Johann
• Created: 06/May/20 12:41
• Updated: 20/May/20 01:44
• Resolved: 20/May/20 01:44

Description

Currently the avro schema gets passed to AvroConversionHelper.createConverterToAvro which itself pocesses passed spark sql DataTypes recursively to resolve structs, arrays, etc. - the AvroSchema gets passed to recursions, but without selection of the relevant field and therefore schema of that field. That leads to a null pointer exception when decimal types will be processed, because in that case the schema of the filed will be retrieved by calling getField on the root schema which is not defined when we deal with nested records.
AvroConversionHelper.scala#L291

The proposed solution is to remove the dependency on the avro schema and derive the particular avro schema for the decimal converter creator case only.

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket
• links to: GitHub Pull Request #1596