Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11502

Map side aggregation is extremely slow

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      For the query as following:

      create table tbl2 as 
      select col1, max(col2) as col2 
      from tbl1 group by col1;
      

      If the column for group by has many different values (for example 400000) and it is in type double, the map side aggregation is very slow. I ran the query which took more than 3 hours , after 3 hours, I have to kill the query.
      The same query can finish in 7 seconds, if I turn off map side aggregation by:

      set hive.map.aggr = false;
      

        Attachments

        1. HIVE-11502.1.patch
          1 kB
          Yongzhi Chen
        2. HIVE-11502.2.patch
          1 kB
          Yongzhi Chen
        3. HIVE-11502.3.patch
          1 kB
          Yongzhi Chen

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              ychena Yongzhi Chen Assign to me
              Reporter:
              ychena Yongzhi Chen

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment