[HUDI-1473] Handle empty parquet files

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: storage-management
• Assignee: Unassigned
• Reporter: Clark Chang
• Created: 18/Dec/20 19:46
• Updated: 12/Mar/22 13:03

Description

Under occasional situations, Hudi bulk insert generates empty parquet files ( I cannot consistently reproduce it, however).

The empty parquet files cause subsequent updates fail due to ParquetUtils trying to read the footer.

In Spark, there is a property: "spark.sql.files.ignoreCorruptFiles" which handles such a case, would Hudi be able to take in this property?