[TAJO-744] (Umbrella) ALTER TABLE ADD/DROP PARTITION statement

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Assignee: JaeHwa Jung
• Reporter: Hyunsik Choi
• Created: 09/Apr/14 08:50
• Updated: 01/Feb/16 01:29

Description

Currently, Tajo does not manage partitioned directly. In Tajo, each partition is just a directory. For each query, a logical planner traverses matched directories in HDFS according to partition predicates.

This approach is not efficient especially in the environment where the number of partitions are very large. It also makes partition management hard.

Tajo should manage partitions directly by using ALTER TABLE ADD/DROP PARTITION statements. A number of partition entries should be stored in the underlying database that catalog uses.

Synopsis of ALTER TABLE ADD/DROP PARTITION

ALTER TABLE table_name [IF NOT EXISTS] ADD COLUMN PARTITION (key1 = 'val2', key2 = 'val2', ...) WITH ('prop_key' = 'prop_val', ...) LOCATION '...';

ALTER TABLE table_name [IF EXISTS] DROP COLUMN PARTITION  (key1 [=|<|<=|>|>=|!=] 'val1', key2 ...,);

Issue Links

• is depended upon by: TAJO-1067 INSERT OVERWRITE INTO should not remove all partitions.
• is related to: TAJO-1253 Catalog API should be queryable
• is related to: TAJO-1176 Implements queryable virtual tables for catalog information

Sub-Tasks

• Add ALTER TABLE ADD/DROP PARTITION statement to parser: Resolved
• Add alter partition method to CatalogStore: Resolved
• Implement logical plan part and DDL executor for alter partition.: Resolved
• Create dynamic partitions to CatalogStore by running insert query or CTAS query.: Resolved
• Apply column partitions to query optimizer.: Resolved
• Make partition pruning based on catalog informations: Resolved
• Implement recover partitions: Resolved
• Add a statement for adding partition to TajoDump: Resolved
• Remove partition_keys table from information_schema: Resolved
• CatalogStore::getAllPartitions need to throw PartitionNotFoundException if there is no partitions on catalog.: Resolved
• When building filtered partition path, check partition directory exists on filesystem.: Resolved
• Disable the alter table add partition statement temporarily.: Resolved
• Allow arbitrary partition path: Open
• Allow date types with IN operator for partition pruning.: Resolved
• PartitionedTableRewriter should use a local variable to remember the volume of partitions.: Resolved
• Implement PartitionFileFragment: Resolved
• Support add multiple partitions in a single ALTER TABLE statement: Open