[HUDI-7331] Test and certify col stats integration with MOR table

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: metadata
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 25/Jan/24 05:08
• Updated: 25/Jan/24 05:08

Description

Lets test and certify col stats integration with MOR table for all operations.

for eg, any write operations (bulk insert, insert, upsert, insert overwrite) should add new entries to col stats index in metadata table. 

rollback: 

for files that were deleted should be removed from col stats (data files). 

for log files added, we should add new entries to col stats 

 

clean: 

any files deleted (data files and log files) should have the entries removed from col stats in MDT. 

 

Similarly, lets also do similar exercise with delete partition and other operations we have with hudi.