[HIVE-15641] Hive/Druid integration: filter on timestamp not pushed to DruidQuery

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Component/s: Druid integration
• Target Version/s: 3.0.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 16/Jan/17 13:11
• Updated: 27/Feb/24 22:23
• Resolved: 17/Feb/18 05:59

Description

It seems we are missing opportunity push Filter operation to DruidQuery.

For instance, for the following query:

EXPLAIN
SELECT i_brand_id, floor_day(`__time`), max(ss_quantity), sum(ss_wholesale_cost) as s
FROM store_sales_sold_time_subset
WHERE floor_day(`__time`) BETWEEN '1999-11-01 00:00:00' AND '1999-11-10 00:00:00'
GROUP BY i_brand_id, floor_day(`__time`)
ORDER BY s
LIMIT 10;
OK
Plan optimized by CBO.

Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)

Stage-0
  Fetch Operator
    limit:10
    Stage-1
      Reducer 3 vectorized
      File Output Operator [FS_17]
        Limit [LIM_16] (rows=1 width=0)
          Number of rows:10
          Select Operator [SEL_15] (rows=1 width=0)
            Output:["_col0","_col1","_col2","_col3"]
          <-Reducer 2 [SIMPLE_EDGE] vectorized
            SHUFFLE [RS_14]
              Group By Operator [GBY_13] (rows=1 width=0)
                Output:["_col0","_col1","_col2","_col3"],aggregations:["max(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0, KEY._col1
              <-Map 1 [SIMPLE_EDGE]
                SHUFFLE [RS_5]
                  PartitionCols:_col0, _col1
                  Group By Operator [GBY_4] (rows=1 width=0)
                    Output:["_col0","_col1","_col2","_col3"],aggregations:["max(_col2)","sum(_col3)"],keys:_col0, _col1
                    Select Operator [SEL_2] (rows=1 width=0)
                      Output:["_col0","_col1","_col2","_col3"]
                      Filter Operator [FIL_12] (rows=1 width=0)
                        predicate:floor_day(__time) BETWEEN '1999-11-01 00:00:00' AND '1999-11-10 00:00:00'
                        TableScan [TS_0] (rows=15888 width=0)
                          tpcds_druid_10@store_sales_sold_time_subset,store_sales_sold_time_subset,Tbl:PARTIAL,Col:NONE,Output:["__time","i_brand_id","ss_quantity","ss_wholesale_cost"],properties:{"druid.query.json":"{\"queryType\":\"select\",\"dataSource\":\"druid_tpcds_ss_sold_time_subset\",\"descending\":false,\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"dimensions\":[\"i_item_id\",\"i_rec_start_date\",\"i_rec_end_date\",\"i_item_desc\",\"i_brand_id\",\"i_brand\",\"i_class_id\",\"i_class\",\"i_category_id\",\"i_category\",\"i_manufact_id\",\"i_manufact\",\"i_size\",\"i_formulation\",\"i_color\",\"i_units\",\"i_container\",\"i_manager_id\",\"i_product_name\",\"c_customer_id\",\"c_salutation\",\"c_first_name\",\"c_last_name\",\"c_preferred_cust_flag\",\"c_birth_day\",\"c_birth_month\",\"c_birth_year\",\"c_birth_country\",\"c_login\",\"c_email_address\",\"c_last_review_date\",\"ca_address_id\",\"ca_street_number\",\"ca_street_name\",\"ca_street_type\",\"ca_suite_number\",\"ca_city\",\"ca_county\",\"ca_state\",\"ca_zip\",\"ca_country\",\"ca_gmt_offset\",\"ca_location_type\",\"s_store_id\",\"s_rec_start_date\",\"s_rec_end_date\",\"s_store_name\",\"s_hours\",\"s_manager\",\"s_market_id\",\"s_geography_class\",\"s_market_desc\",\"s_market_manager\",\"s_division_id\",\"s_division_name\",\"s_company_id\",\"s_company_name\",\"s_street_number\",\"s_street_name\",\"s_street_type\",\"s_suite_number\",\"s_city\",\"s_county\",\"s_state\",\"s_zip\",\"s_country\",\"s_gmt_offset\"],\"metrics\":[\"ss_ticket_number\",\"ss_quantity\",\"ss_wholesale_cost\",\"ss_list_price\",\"ss_sales_price\",\"ss_ext_discount_amt\",\"ss_ext_sales_price\",\"ss_ext_wholesale_cost\",\"ss_ext_list_price\",\"ss_ext_tax\",\"ss_coupon_amt\",\"ss_net_paid\",\"ss_net_paid_inc_tax\",\"ss_net_profit\",\"i_current_price\",\"i_wholesale_cost\",\"s_number_employees\",\"s_floor_space\",\"s_tax_precentage\"],\"granularity\":\"all\",\"pagingSpec\":{\"threshold\":16384},\"context\":{\"druid.query.fetch\":false}}","druid.query.type":"select"}