[IMPALA-1839] Introducing virtual columns for auto partition pruning

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Impala 2.1.1
• Labels: partitions
• Assignee: Unassigned
• Reporter: Daniel Lanza García
• Created: 04/Mar/15 11:57
• Updated: 10/Mar/15 05:56
• Resolved: 10/Mar/15 05:56

Description

It is my first message in the community and I don't know if I should write here directly.

In the organisation where I work, we have developed a new feature for impala that we needed which consists you don't have to specify the partitioning columns in your queries in order to prune partitions. I don't know if it is useful for the project but for us is a good advantage and is working very well.

To create a table which use this feature, just is necessary name the partitioning columns with a specific format. The format is the following: COLUMN_part_FUNCTION[_ARGS] where:

• COLUMN is the name of the normal column which use for pruning
• part, mandatory subfix to use this feature
• FUNCTION is the name of the function to apply to the COLUMN (e.g. year, month, day, module, ...)
• [_ARGS] some functions (as module) need some extra arguments, these should be placed at the end of the name.

You can find the implementation here: https://github.com/dlanza1/Impala/tree/cdh5-2.1.1_5.3.1

Sorry for disturb and thanks a lot for your reading.