[SPARK-5302] Add support for SQLContext "partition" columns

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SQL
• Assignee: Cheng Lian
• Reporter: Bob Tiernay
• Created: 18/Jan/15 00:24
• Updated: 16/Sep/15 08:06
• Resolved: 15/Sep/15 21:42

Description

For SQLContext (not HiveContext) it would be very convenient to support a virtual column that maps to part of the the file path, similar to what is done in Hive for partitions (e.g. /data/clicks/dt=2015-01-01/ where dt is a column of type TEXT).

The API could allow the user to type the column using an appropriate DataType instance. This new field could be addressed in SQL statements much the same as is done in Hive.

As a consequence, pruning of partitions could be possible when executing a query and also remove the need to materialize a column in each logical partition that is already encoded in the path name. Furthermore, this would provide an nice interop and migration strategy for Hive users who may one day use SQLContext directly.

Issue Links

• is related to: SPARK-5182 Partitioning support for tables created by the data source API