Hive
  1. Hive
  2. HIVE-582

"SELECT count(1) as cnt from src x" not working for column pruning

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The query "SELECT count(1) as cnt from src x" has some problems, even though nullgroup*.q passed in tests.
      Will check and solve it in this jira.
      A wrong plan sometimes can be generated due to some unknow factor:
      Wrong plan:

      STAGE PLANS:
        Stage: Stage-1
          Map Reduce
            Alias -> Map Operator Tree:
              x 
                  Select Operator
                    Reduce Output Operator
                      sort order: 
                      tag: -1
            Reduce Operator Tree:
              Group By Operator
                aggregations:
                      expr: count(VALUE._col0)
                mode: complete
                Select Operator
                  expressions:
                        expr: _col0
                        type: bigint
                  File Output Operator
                    compressed: false
                    GlobalTableId: 0
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
      
        Stage: Stage-0
          Fetch Operator
            limit: -1
      

      Without unknow factor, correct plan can be generated:

      STAGE PLANS:
        Stage: Stage-1
          Map Reduce
            Alias -> Map Operator Tree:
              x 
                  Select Operator
                    Group By Operator
                      aggregations:
                            expr: count(1)
                      mode: hash
                      Reduce Output Operator
                        sort order: 
                        tag: -1
                        value expressions:
                              expr: _col0
                              type: bigint
            Reduce Operator Tree:
              Group By Operator
                aggregations:
                      expr: count(VALUE._col0)
                mode: mergepartial
                Select Operator
                  expressions:
                        expr: _col0
                        type: bigint
                  File Output Operator
                    compressed: false
                    GlobalTableId: 0
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
      

        Activity

        He Yongqiang created issue -
        Hide
        He Yongqiang added a comment -

        It seems that the wrong plan is generated when setting hive.map.aggr=false, and the reduce output nothing(because of the column pruning). Reducer got nothing, so the exception occured.

        Show
        He Yongqiang added a comment - It seems that the wrong plan is generated when setting hive.map.aggr=false, and the reduce output nothing(because of the column pruning). Reducer got nothing, so the exception occured.
        He Yongqiang made changes -
        Field Original Value New Value
        Attachment hive-582-2009-06-25.patch [ 12411811 ]
        Zheng Shao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Namit Jain added a comment -

        looks good

        Show
        Namit Jain added a comment - looks good
        Hide
        Namit Jain added a comment -

        Committed. Thanks Yongqiang

        Show
        Namit Jain added a comment - Committed. Thanks Yongqiang
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.4.0 [ 12313714 ]
        Resolution Fixed [ 1 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4d 9h 23m 1 Zheng Shao 29/Jun/09 20:00
        Patch Available Patch Available Resolved Resolved
        5h 54m 1 Namit Jain 30/Jun/09 01:54
        Resolved Resolved Closed Closed
        899d 23h 12m 1 Carl Steinbach 17/Dec/11 00:07

          People

          • Assignee:
            He Yongqiang
            Reporter:
            He Yongqiang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development