[HUDI-7556] Fix MDT validator to account for additional partitions in MDT

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0, 1.0.0
• Component/s: metadata
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 29/Mar/24 16:12
• Updated: 13/Apr/24 20:04
• Resolved: 13/Apr/24 20:04

Description

There is a chance that MDT could list additional partitions when compared to FS based listing. 

reason is: 

We load active timeline from metaclient and poll FS based listing for completed commits. And then we poll MDT for list of all partitions. in between these two, there could be a commit that could have been completed and hence MDT could be serving that as well. So, lets account for that in our validation tool 

Issue Links

• links to: GitHub Pull Request #10939
• links to: GitHub Pull Request #10985
• links to: GitHub Pull Request #10986