Blocker Description
Out of the box, we generate col stats only for top level fields. but user does have an option to overide the columns for which they need hudi to generate cols stats for.
When we tested for a nested field, we realized that we have a gap here. Hudi does generate col stats for base files properly even for nested fields. but log files are missing to generate col stats.
The linked code snippet will only honor top level fields.
So, we have two fixes here.
Fix1: lets avoid generating stats even for base files. also throw exception if someone explicitly sets a nested field with "hoodie.metadata.index.column.stats.column.list".
Fix2: Follow up to support nested field col stats generation.
Fix1 is a blocker for 1.0 release. May be we can punt fix 2 for later.