[HUDI-3189] Fallback to full table scan with incremental query when files are cleaned up or achived for MOR table

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: incremental-query
• Labels: pull-request-available sev:high
• Story Points: 2
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 06/Jan/22 21:20
• Updated: 10/Mar/23 01:03

Description

https://github.com/apache/hudi/pull/3946

was added to fallback to full table scan with incremental query on 2 occasions:

1. files are cleaned up, but active timeline still returns the commits.
2. commits are archived. 

 

There are two follow ups from the original PR. 

a. fs.isExists() call should be routed to metadata table. 

b. Add similar support to MOR table. 

 

Issue Links

• duplicates: HUDI-4410 Incremental read from archived commits for Spark
• is duplicated by: HUDI-4509 FileNotFoundException during doing readStream
• links to: GitHub Pull Request #6141