Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12535

Dynamic Hash Join: Key references are cyclic

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.0.0
    • None
    • Query Planning
    • None

    Description

      MAPJOIN_4227 is inside "Reducer 2", but refers back to "Reducer 2" in its keys. It should say "Map 1" there.

      |                |<-Reducer 2 [SIMPLE_EDGE] vectorized, llap                                                                                                                                                                                                        |
      |                   Reduce Output Operator [RS_4189]                                                                                                                                                                                                                |
      |                      key expressions:_col0 (type: string), _col1 (type: int)                                                                                                                                                                                      |
      |                      Map-reduce partition columns:_col0 (type: string), _col1 (type: int)                                                                                                                                                                         |
      |                      sort order:++                                                                                                                                                                                                                                |
      |                      Statistics:Num rows: 83 Data size: 9213 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                                         |
      |                      value expressions:_col2 (type: double)                                                                                                                                                                                                       |
      |                      Group By Operator [OP_4229]                                                                                                                                                                                                                  |
      |                         aggregations:["sum(_col2)"]                                                                                                                                                                                                               |
      |                         keys:_col0 (type: string), _col1 (type: int)                                                                                                                                                                                              |
      |                         outputColumnNames:["_col0","_col1","_col2"]                                                                                                                                                                                               |
      |                         Statistics:Num rows: 83 Data size: 9213 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                                      |
      |                         Select Operator [OP_4228]                                                                                                                                                                                                                 |
      |                            outputColumnNames:["_col0","_col1","_col2"]                                                                                                                                                                                            |
      |                            Statistics:Num rows: 166 Data size: 26394 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                                 |
      |                            Map Join Operator [MAPJOIN_4227]                                                                                                                                                                                                       |
      |                            |  condition map:[{"":"Inner Join 0 to 1"}]                                                                                                                                                                                            |
      |                            |  keys:{"Reducer 2":"KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: int)","Map 5":"KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: int)"}  |
      |                            |  outputColumnNames:["_col1","_col3","_col5"]                                                                                                                                                                                         |
      |                            |  Statistics:Num rows: 166 Data size: 26394 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                              |
      |                            |<-Map 5 [CUSTOM_SIMPLE_EDGE] vectorized, llap                                                                                                                                                                                         |
      |                            |  Reduce Output Operator [RS_4226]                                                                                                                                                                                                    |
      |                            |     key expressions:_col1 (type: bigint), year(_col2) (type: int), month(_col2) (type: int)                                                                                                                                          |
      |                            |     Map-reduce partition columns:_col1 (type: bigint), year(_col2) (type: int), month(_col2) (type: int)                                                                                                                             |
      |                            |     sort order:+++                                                                                                                                                                                                                   |
      |                            |     Statistics:Num rows: 74973886 Data size: 5098224248 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                 |
      |                            |     value expressions:_col0 (type: float), _col2 (type: date)                                                                                                                                                                        |
      |                            |     Select Operator [OP_4225]                                                                                                                                                                                                        |
      |                            |        outputColumnNames:["_col0","_col1","_col2"]                                                                                                                                                                                   |
      |                            |        Statistics:Num rows: 74973886 Data size: 5098224248 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                              |
      |                            |        Filter Operator [FIL_4224]                                                                                                                                                                                                    |
      |                            |           predicate:((account_id is not null and month(effective_date) BETWEEN 4 AND 7) and month(effective_date) is not null) (type: boolean)                                                                                       |
      |                            |           Statistics:Num rows: 74973886 Data size: 5098224248 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                           |
      |                            |           TableScan [TS_4171]                                                                                                                                                                                                        |
      |                            |              alias:t                                                                                                                                                                                                                 |
      |                            |              Statistics:Num rows: 149947772 Data size: 10196448496 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                      |
      |                            |<-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap                                                                                                                                                                                         |
      |                               Reduce Output Operator [RS_4223]                                                                                                                                                                                                    |
      |                                  key expressions:_col0 (type: bigint), year(_col2) (type: int), month(_col2) (type: int)                                                                                                                                          |
      |                                  Map-reduce partition columns:_col0 (type: bigint), year(_col2) (type: int), month(_col2) (type: int)                                                                                                                             |
      |                                  sort order:+++                                                                                                                                                                                                                   |
      |                                  Statistics:Num rows: 50289673 Data size: 8197216699 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                 |
      |                                  value expressions:_col1 (type: string)                                                                                                                                                                                           |
      |                                  Map Join Operator [MAPJOIN_4222]                                                                                                                                                                                                 |
      |                                  |  condition map:[{"":"Left Semi Join 0 to 1"}]                                                                                                                                                                                  |
      |                                  |  keys:{"Map 1":"_col1 (type: string)","Map 4":"_col0 (type: string)"}                                                                                                                                                          |
      |                                  |  outputColumnNames:["_col0","_col1","_col2"]                                                                                                                                                                                   |
      |                                  |  Statistics:Num rows: 50289673 Data size: 8197216699 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                              |
      |                                  |<-Map 4 [BROADCAST_EDGE] vectorized, llap                                                                                                                                                                                       |
      |                                  |  Reduce Output Operator [RS_4179]                                                                                                                                                                                              |
      |                                  |     key expressions:_col0 (type: string)                                                                                                                                                                                       |
      |                                  |     Map-reduce partition columns:_col0 (type: string)                                                                                                                                                                          |
      |                                  |     sort order:+                                                                                                                                                                                                               |
      |                                  |     Statistics:Num rows: 1 Data size: 99 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                          |
      |                                  |     Group By Operator [OP_4219]                                                                                                                                                                                                |
      |                                  |        keys:_col0 (type: string)                                                                                                                                                                                               |
      |                                  |        outputColumnNames:["_col0"]                                                                                                                                                                                             |
      |                                  |        Statistics:Num rows: 1 Data size: 99 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                       |
      |                                  |        Select Operator [OP_4218]                                                                                                                                                                                               |
      |                                  |           outputColumnNames:["_col0"]                                                                                                                                                                                          |
      |                                  |           Statistics:Num rows: 3 Data size: 297 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                   |
      |                                  |           Filter Operator [FIL_4217]                                                                                                                                                                                           |
      |                                  |              predicate:(account_type = 'order ahead') (type: boolean)                                                                                                                                                      |
      |                                  |              Statistics:Num rows: 3 Data size: 294 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                |
      |                                  |              TableScan [TS_4168]                                                                                                                                                                                               |
      |                                  |                 alias:at                                                                                                                                                                                                       |
      |                                  |                 Statistics:Num rows: 13 Data size: 1274 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                           |
      |                                  |<-Select Operator [OP_4221]                                                                                                                                                                                                     |
      |                                        outputColumnNames:["_col0","_col1","_col2"]                                                                                                                                                                                |
      |                                        Statistics:Num rows: 50289673 Data size: 8197216699 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                           |
      |                                        Filter Operator [FIL_4220]                                                                                                                                                                                                 |
      |                                           predicate:(((account_id is not null and (account_type = 'order ahead')) and year(effective_date) is not null) and month(effective_date) is not null) (type: boolean)                                                |
      |                                           Statistics:Num rows: 50289673 Data size: 8197216699 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                        |
      |                                           TableScan [TS_4165]                                                                                                                                                                                                     |
      |                                              alias:a                                                                                                                                                                                                              |
      |                                              Statistics:Num rows: 201158695 Data size: 32788867285 Basic stats: COMPLETE Column stats: COMPLETE                                                                                                                                                                                                                                                            
      

      Attachments

        1. philz_26.txt
          29 kB
          Gopal Vijayaraghavan

        Activity

          People

            jdere Jason Dere
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: