[CARBONDATA-1377] Implement hive partition

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hive-integration
• Assignee: cen yuhai
• Reporter: cen yuhai
• Created: 13/Aug/17 12:48
• Updated: 15/Oct/17 05:38

Description

Current partition implement is like database, If I want to use carbon to replace parquet massively, we must make the usage of carbon the same with parquet/orc.

Hive users should able to switch to CarbonData for all the new partitions being created. Hive support format to be specified at partition level.
Example:

create table rtestpartition (col1 string, col2 int) partitioned by (col3 int) stored as parquet;
insert into rtestpartition partition(col3=10) select "pqt", 1;
insert into rtestpartition partition(col3=20) select "pqt", 1;
insert into rtestpartition partition(col3=10) select "pqt", 1;
insert into rtestpartition partition(col3=20) select "pqt", 1;

hive creates folder like
    /db1/table1/col3=10/0001_file.pqt
    /db1/table1/col3=10/0002_file.pqt
    /db1/table1/col3=20/0001_file.pqt
    /db1/table1/col3=20/0002_file.pqt

Hive users can now change new partitions to CarbonData, how ever old partitions still be with parquet and require migration scripts to move to CarbonData format.

alter table rtestpartition set fileformat carbondata;

insert into rtestpartition partition(col3=30) select "cdata", 1;
insert into rtestpartition partition(col3=40) select "cdata", 1;

hive creates folder like
    /db1/table1/col3=10/0001_file.pqt
    /db1/table1/col3=10/0002_file.pqt
    /db1/table1/col3=20/0001_file.pqt
    /db1/table1/col3=20/0002_file.pqt
    /db1/table1/col3=30/<carbondatafiles>
    /db1/table1/col3=40/<carbondatafiles>