[HUDI-8556] Trim the number of columns to generate col stats out of the box

• Type: Improvement
• Status: Patch Available
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Component/s: dataskipping, metadata
• Labels: pull-request-available
• Assignee: Jonathan Vexler
• Reporter: sivabalan narayanan

Reviewers

sivabalan narayanan

• Created: 3 days ago 20:13
• Updated: Yesterday 22:15

Description

As of now, out of the box, we generate col stats for all top level fields. This could be prohibitively expensive for wider tables having 1000 columns. So, we should trim it down to say first 32 to level columns for a good out of the box performance. 

Users will anyway have an option to override if need be. 

 

Lets add a config to drive this, and out of the box we can set it to 32. 

 

Issue Links

• links to: GitHub Pull Request #12310