[HUDI-2013] Fallback to file listing may lead to data loss

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Labels: pull-request-available
• Assignee: Prashant Wason
• Reporter: Prashant Wason
• Created: 15/Jun/21 05:59
• Updated: 01/Jul/21 19:53
• Resolved: 01/Jul/21 19:53

Description

When fallback to file listing mode is enabled (hoodie.metadata.fallback.enable, default is true), then if listing from the metadata table leads to an exception the normal file-system listing used.

Metadata table listing may fail if the table is inconsistent or due to bugs. Falling back to file listing has the following downsides:

1. It masks the issue as the commit does not fail (only an exception is logged).
2. By the time the issue is discovered, logs may have been lost
3. There is no guarantee that all the commits wrote/updated the correct files.

Since listing from metadata table is per-partition, the issue is further complicated when listing for some partitions succeeds (file-list retrieved from metadata table) and fails for other partitions (file list retrieved from filesystem).

 

Issue Links

• is a child of: HUDI-1292 [Umbrella] RFC-15 : Metadata Table for File Listing and other table metadata
• links to: GitHub Pull Request #3079