[HUDI-2550] Add support to configure no of small files to consider with MOR

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 0.10.0
• Component/s: writer-core
• Labels: pull-request-available sev:critical user-support-issues
• Assignee: Alexey Kudinkin
• Reporter: sivabalan narayanan
• Created: 13/Oct/21 05:55
• Updated: 08/Dec/21 07:00

Description

Looks like in MOR, when an index is used which cannot index log files (which is the case for all out of box indexes in hudi), we just choose the smallest parquet file for every commit. So, over time, every file will grow to become fullest is the idea here. In other words, only one small file will be bin backed per commit even though there could be more. 

source link

 

We can add a config which can control the total number of files considered as small files for MOR table when index which cannot index log files are used. 

We can leave the default value to 1 (current behavior). But for interested users, this should be flexible. 

 

Original issue

https://github.com/apache/hudi/issues/3676 

 

 

Issue Links

• links to: GitHub Pull Request #3986