Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5945

ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Here is an example

      select
         i_item_id,
         s_state,
         avg(ss_quantity) agg1,
         avg(ss_list_price) agg2,
         avg(ss_coupon_amt) agg3,
         avg(ss_sales_price) agg4
      FROM store_sales
      JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
      JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
      JOIN customer_demographics on (store_sales.ss_cdemo_sk = customer_demographics.cd_demo_sk)
      JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
      where
         cd_gender = 'F' and
         cd_marital_status = 'U' and
         cd_education_status = 'Primary' and
         d_year = 2002 and
         s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
      group by
         i_item_id,
         s_state
      order by
         i_item_id,
         s_state
      limit 100;
      

      I turned off noconditionaltask. So, I expected that there will be 4 Map-only jobs for this query. However, I got 1 Map-only job (joining strore_sales and date_dim) and 3 MR job (for reduce joins.)

      So, I checked the conditional task determining the plan of the join involving item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask, aliasToFileSizeMap contains all input tables used in this query and the intermediate table generated by joining store_sales and date_dim. So, when we sum the size of all small tables, the size of store_sales (which is around 45GB in my test) will be also counted.

        Attachments

        1. HIVE-5945.1.patch.txt
          14 kB
          Navis Ryu
        2. HIVE-5945.2.patch.txt
          14 kB
          Navis Ryu
        3. HIVE-5945.3.patch.txt
          16 kB
          Navis Ryu
        4. HIVE-5945.4.patch.txt
          15 kB
          Navis Ryu
        5. HIVE-5945.5.patch.txt
          19 kB
          Navis Ryu
        6. HIVE-5945.6.patch.txt
          20 kB
          Navis Ryu
        7. HIVE-5945.7.patch.txt
          21 kB
          Navis Ryu
        8. HIVE-5945.8.patch.txt
          22 kB
          Navis Ryu

          Issue Links

            Activity

              People

              • Assignee:
                navis Navis Ryu
                Reporter:
                yhuai Yin Huai
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: