[PARQUET-1784] Column-wise configuration

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: parquet-mr
• Labels: pull-request-available
• Assignee: Gabor Szadovszky
• Reporter: Gabor Szadovszky
• Created: 05/Feb/20 10:19
• Updated: 23/Jun/24 03:31
• Resolved: 26/Feb/20 11:39

Description

After adding some new statistics and encodings into Parquet it is getting very hard to be smart and choose the best configs automatically. For example for which columns should we save column index and/or bloom-filters? Is it worth using dictionary for a column that we know will fall back to another encoding?

The idea of this feature is to allow the library user to fine-tune the configuration by setting it column-wise. To support this we extend the existing configuration keys by a suffix to identify the related column. (From now on we introduce new keys following the same syntax.)
{key of the configuration}#{column path in the file schema}
For example: parquet.enable.dictionary#column.path.col_1

This jira covers the framework to support the column-wise configuration with the implementation of some existing configs where it make sense (e.g. parquet.enable.dictionary). Implementing new configuration is not part of this effort.

Issue Links

• relates to: PARQUET-1805 Refactor the configuration for bloom filters
• links to: GitHub Pull Request #754