[HUDI-6206] Enhance Metadata tests to validate col stats

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: metadata
• Labels: release-0.14.0-blocker
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 12/May/23 19:07
• Updated: 12/May/23 19:11

Description

I happened to check for col stats tests in TestHoodieBackedMetadata and did not find any good tests that validates col stats records from MDT and bloom partition records. I do see some validations for log files, but none for base files. we do have one test named testColStatsPrefixLookup which tests col stats. but if we can add a separate validateColStats() method we can call it with validateMetadata() for any tests we have to gain more coverage. 

 

We do a sparate test class TestColumnStatsIndex for testing col stats. But would be good to enhance the validations in TestHoodieBackedMetadata tests.

we should enhance them for sure.