[SPARK-27858] Fix for avro deserialization on union types with multiple non-null types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.3, 3.0.0
• Fix Version/s: 2.4.4, 3.0.0
• Component/s: SQL
• Assignee: Gabbi Merz
• Reporter: Dongjoon Hyun
• Created: 28/May/19 00:09
• Updated: 28/May/19 15:39
• Resolved: 28/May/19 03:23

Description

This issue aims to fix a union avro type with more than one non-null value (for instance `["string", "null", "int"]`) the deserialization to a DataFrame would throw a `java.lang.ArrayIndexOutOfBoundsException`. The issue was that the `fieldWriter` relied on the index from the avro schema before null were filtered out.

Issue Links

• links to: GitHub Pull Request #24722