[DRILL-468] Support for FileSystem partitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Steven Phillips
• Reporter: Steven Phillips
• Created: 28/Mar/14 08:51
• Updated: 31/Jul/14 06:31
• Resolved: 23/Apr/14 03:19

Description

For filesystem partitioning, we want to use the existing directory structure of the data. So, if a selection is a directory that contains subdirectories, the name of the directory a given record was stored in can be included as a field in that record. For example, given this structure:

/data
/a
file.csv
/b
file.csv

select * from dfs.`/data`

will include a column named dir0, with possible values a and b. This can be extended to a hierarchy of partitions. For example,

/data
/a
/1
file.csv
/2
file.csv
/b
file.csv

would have columns dir0 (with possible values a and b) and dir1 (with possible values 1, 2 and null).

The data type will always be VARCHAR for the partition columns.