Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11502

Map side aggregation is extremely slow

    Details

      Description

      For the query as following:

      create table tbl2 as 
      select col1, max(col2) as col2 
      from tbl1 group by col1;
      

      If the column for group by has many different values (for example 400000) and it is in type double, the map side aggregation is very slow. I ran the query which took more than 3 hours , after 3 hours, I have to kill the query.
      The same query can finish in 7 seconds, if I turn off map side aggregation by:

      set hive.map.aggr = false;
      

        Attachments

        1. HIVE-11502.1.patch
          1 kB
          Yongzhi Chen
        2. HIVE-11502.2.patch
          1 kB
          Yongzhi Chen
        3. HIVE-11502.3.patch
          1 kB
          Yongzhi Chen

          Issue Links

            Activity

              People

              • Assignee:
                ychena Yongzhi Chen
                Reporter:
                ychena Yongzhi Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: