[SPARK-21852] Empty Parquet Files created as a result of spark jobs fail when read

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.2.0
• Component/s: Input/Output
• External issue URL: https://issues.apache.org/jira/browse/PARQUET-1080
• Assignee: Unassigned
• Reporter: Shivam Dalmia
• Created: 28/Aug/17 16:41
• Updated: 12/Dec/22 18:10
• Resolved: 01/Sep/17 00:46

Description

I have faced an issue intermittently with certain spark jobs writing parquet files which apparently succeed but the written .parquet directory in HDFS is an empty directory (with no _SUCCESS and _metadata parts, even). Surprisingly, no errors are thrown from spark dataframe writer.

However, when attempting to read this written file, spark throws the error:
Unable to infer schema for Parquet. It must be specified manually