[SPARK-46633] Reading a non-empty Avro file with empty blocks returns 0 records

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Ivan Sadikov
• Reporter: Ivan Sadikov
• Created: 09/Jan/24 04:25
• Updated: 22/Mar/24 00:32
• Resolved: 10/Jan/24 00:47

Description

When an Avro file contains empty blocks, Spark returns 0 records while "fastavro" and "avro-python-3" both read the file correctly and return records.

 

This is due to the way Spark handles empty blocks (or does not handle). Call to `hasNext` loads the next block and if that block is empty, it returns false. But instead of exiting the loop, we need to probe the next block until sync point is reached.

Issue Links

• links to: GitHub Pull Request #44635