Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4240

optimize hive.enforce.bucketing and hive.enforce sorting insert

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.11.0
    • Query Processor
    • None
    • Reviewed

    Description

      Consider the following scenario:

      set hive.optimize.bucketmapjoin = true;
      set hive.optimize.bucketmapjoin.sortedmerge = true;
      set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
      set hive.enforce.bucketing=true;
      set hive.enforce.sorting=true;
      set hive.exec.reducers.max = 1;
      set hive.merge.mapfiles=false;
      set hive.merge.mapredfiles=false;

      – Create two bucketed and sorted tables
      CREATE TABLE test_table1 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;
      CREATE TABLE test_table2 (key INT, value STRING) PARTITIONED BY (ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS;

      FROM src
      INSERT OVERWRITE TABLE test_table1 PARTITION (ds = '1') SELECT *;

      – Insert data into the bucketed table by selecting from another bucketed table
      – This should be a map-only operation
      INSERT OVERWRITE TABLE test_table2 PARTITION (ds = '1')
      SELECT a.key, a.value FROM test_table1 a WHERE a.ds = '1';

      We should not need a reducer to perform the above operation.

      Attachments

        1. hive.4240.1.patch
          71 kB
          Namit Jain
        2. hive.4240.2.patch
          174 kB
          Namit Jain
        3. hive.4240.3.patch
          175 kB
          Namit Jain
        4. hive.4240.4.patch
          175 kB
          Namit Jain
        5. hive.4240.5.patch
          176 kB
          Namit Jain
        6. hive.4240.5.patch-nohcat
          183 kB
          Namit Jain

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            namit Namit Jain Assign to me
            namit Namit Jain
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment