When fallback to file listing mode is enabled (hoodie.metadata.fallback.enable, default is true), then if listing from the metadata table leads to an exception the normal file-system listing used.
Metadata table listing may fail if the table is inconsistent or due to bugs. Falling back to file listing has the following downsides:
- It masks the issue as the commit does not fail (only an exception is logged).
- By the time the issue is discovered, logs may have been lost
- There is no guarantee that all the commits wrote/updated the correct files.
Since listing from metadata table is per-partition, the issue is further complicated when listing for some partitions succeeds (file-list retrieved from metadata table) and fails for other partitions (file list retrieved from filesystem).