[HUDI-4022] Add support to validate table's internal state with integ test infra

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tests-ci
• Story Points: 1
• Epic Link: CI tests improvements
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 02/May/22 20:46
• Updated: 10/Mar/23 01:50

Description

we might need to add validation to integ test suite to test data and metadata table's internal state or steady state. For eg, need to validate that

cleaning has progressed as per the configs.

Archival has progressed as per the configs. archival has archived the timeline files. 

if metadata is enabled, metadata table compaction progressed as per the configs. 

if metadata is enabled, metadata table cleaning has progressed as per the configs. 

if metadata is enabled, metadata table archival has progressed and has archived the timeline files.