Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4665

Partition pruning not working for hive partitioned table with 'LIKE' and '=' filter

    XMLWordPrintableJSON

    Details

      Description

      This problem was initially reported by Shankar Mane <shankar.mane@games24x7.com>

      > Problem:
      >
      > 1. In drill, we are using hive partition table. But explain plan (same
      > query) for like and = operator differs and used all partitions in case of
      > like operator.
      > 2. If you see below drill explain plans: Like operator uses all
      > partitions where
      > = operator uses only partition filtered by log_date condition.

      I reproduced the reported issue. I have a partitioned hive external table with the following schema:

      create external table if not exists lineitem_partitioned (
          l_orderkey bigint,
          l_partkey bigint,
          l_suppkey bigint,
          l_linenumber bigint,
          l_quantity double,
          l_extendedprice double,
          l_discount double,
          l_tax double,
          l_returnflag string,
          l_linestatus string,
          l_shipdate string,
          l_commitdate string,
          l_receiptdate string,
          l_shipinstruct string,
          l_shipmode string,
          l_comment string
      )
      partitioned by (year int, month int, day int)
      STORED AS PARQUET
      LOCATION '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem';
      
      alter table lineitem_partitioned add partition(year=2015, month=1, day=1) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=2) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=3) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=4) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=5) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=6) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=7) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=8) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=9) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9';
      alter table lineitem_partitioned add partition(year=2015, month=1, day=10) location  '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10';
      

      Without 'LIKE', drill plans right:

      0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 where `day` = 2 limit 10;
      +------+------+
      | text | json |
      +------+------+
      | 00-00    Screen
      00-01      Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], day=[$3])
      00-02        SelectionVectorRemover
      00-03          Limit(fetch=[10])
      00-04            Limit(fetch=[10])
      00-05              Project(l_shipdate=[$1], l_linestatus=[$0], l_shipinstruct=[$2], day=[$3])
      00-06                Scan(groupscan=[HiveScan [table=Table(dbName:md815_db2k, tableName:lineitem_partitioned_db2k_2_999), columns=[`l_linestatus`, `l_shipdate`, `l_shipinstruct`, `day`], numPartitions=1, partitions= [Partition(values:[2015, 1, 2])], inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2]]])
      

      With 'LIKE', pruning is not happening:

      0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 where `day` = 2 and l_shipinstruct like '%BACK%' limit 10;
      +------+------+
      | text | json |
      +------+------+
      | 00-00    Screen
      00-01      Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], day=[$3])
      00-02        SelectionVectorRemover
      00-03          Limit(fetch=[10])
      00-04            Limit(fetch=[10])
      00-05              Project(l_shipdate=[$1], l_linestatus=[$0], l_shipinstruct=[$2], day=[$3])
      00-06                SelectionVectorRemover
      00-07                  Filter(condition=[AND(=($3, 2), LIKE($2, '%BACK%'))])
      00-08                    Scan(groupscan=[HiveScan [table=Table(dbName:md815_db2k, tableName:lineitem_partitioned_db2k_2_999), columns=[`l_linestatus`, `l_shipdate`, `l_shipinstruct`, `day`], numPartitions=10, partitions= [Partition(values:[2015, 1, 1]), Partition(values:[2015, 1, 10]), Partition(values:[2015, 1, 2]), Partition(values:[2015, 1, 3]), Partition(values:[2015, 1, 4]), Partition(values:[2015, 1, 5]), Partition(values:[2015, 1, 6]), Partition(values:[2015, 1, 7]), Partition(values:[2015, 1, 8]), Partition(values:[2015, 1, 9])], inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8, maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9]]])
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gparai Gautam Parai
                Reporter:
                cchang@maprtech.com Chun Chang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: