[HIVE-4240] optimize hive.enforce.bucketing and hive.enforce sorting insert

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 28/Mar/13 04:41
• Updated: 28/Mar/16 08:21
• Resolved: 03/Apr/13 07:05

Description

Consider the following scenario:

set hive.optimize.bucketmapjoin = true;
set hive.optimize.bucketmapjoin.sortedmerge = true;
set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
set hive.enforce.bucketing=true;
set hive.enforce.sorting=true;
set hive.exec.reducers.max = 1;
set hive.merge.mapfiles=false;
set hive.merge.mapredfiles=false;

– Create two bucketed and sorted tables
CREATE TABLE test_table1 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;
CREATE TABLE test_table2 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;

FROM src
INSERT OVERWRITE TABLE test_table1 PARTITION (ds = '1') SELECT *;

– Insert data into the bucketed table by selecting from another bucketed table
– This should be a map-only operation
INSERT OVERWRITE TABLE test_table2 PARTITION (ds = '1')
SELECT a.key, a.value FROM test_table1 a WHERE a.ds = '1';

We should not need a reducer to perform the above operation.

Issue Links

• relates to: HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join