Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4240

optimize hive.enforce.bucketing and hive.enforce sorting insert

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Consider the following scenario:

      set hive.optimize.bucketmapjoin = true;
      set hive.optimize.bucketmapjoin.sortedmerge = true;
      set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
      set hive.enforce.bucketing=true;
      set hive.enforce.sorting=true;
      set hive.exec.reducers.max = 1;
      set hive.merge.mapfiles=false;
      set hive.merge.mapredfiles=false;

      – Create two bucketed and sorted tables
      CREATE TABLE test_table1 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;
      CREATE TABLE test_table2 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;

      FROM src
      INSERT OVERWRITE TABLE test_table1 PARTITION (ds = '1') SELECT *;

      – Insert data into the bucketed table by selecting from another bucketed table
      – This should be a map-only operation
      INSERT OVERWRITE TABLE test_table2 PARTITION (ds = '1')
      SELECT a.key, a.value FROM test_table1 a WHERE a.ds = '1';

      We should not need a reducer to perform the above operation.

        Attachments

        1. hive.4240.5.patch-nohcat
          183 kB
          Namit Jain
        2. hive.4240.5.patch
          176 kB
          Namit Jain
        3. hive.4240.4.patch
          175 kB
          Namit Jain
        4. hive.4240.3.patch
          175 kB
          Namit Jain
        5. hive.4240.2.patch
          174 kB
          Namit Jain
        6. hive.4240.1.patch
          71 kB
          Namit Jain

          Issue Links

            Activity

              People

              • Assignee:
                namit Namit Jain
                Reporter:
                namit Namit Jain
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: