Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Lets test and certify col stats integration with MOR table for all operations.
for eg, any write operations (bulk insert, insert, upsert, insert overwrite) should add new entries to col stats index in metadata table.
rollback:
for files that were deleted should be removed from col stats (data files).
for log files added, we should add new entries to col stats
clean:
any files deleted (data files and log files) should have the entries removed from col stats in MDT.
Similarly, lets also do similar exercise with delete partition and other operations we have with hudi.