[DRILL-4071] Partition pruning fails when a Coalesce() function appears with partition filter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: Query Planning & Optimization
• Assignee: Mehant Baid
• Reporter: Aman Sinha
• Reviewer: Rahul Kumar Challapalli
• Created: 11/Nov/15 02:15
• Updated: 14/Dec/15 22:25
• Resolved: 12/Nov/15 07:32

Description

Pruning fails for this query:

0: jdbc:drill:zk=local> explain plan for select count(*) from dfs.`/Users/asinha/data/multilevel/parquet` where dir0 = 1994 and coalesce(o_clerk, 'Clerk') = '';
+------+------+
| text | json |
+------+------+
| 00-00    Screen
00-01      Project(EXPR$0=[$0])
00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()])
00-03          Project($f0=[0])
00-04            SelectionVectorRemover
00-05              Filter(condition=[AND(=($0, 1994), =(CASE(IS NOT NULL($1), $1, 'Clerk'), ''))])
00-06                Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1994/Q1/orders_94_q1.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1994/Q2/orders_94_q2.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1994/Q3/orders_94_q3.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1994/Q4/orders_94_q4.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1995/Q1/orders_95_q1.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1995/Q2/orders_95_q2.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1995/Q3/orders_95_q3.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1995/Q4/orders_95_q4.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1996/Q1/orders_96_q1.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1996/Q2/orders_96_q2.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1996/Q3/orders_96_q3.parquet], ReadEntryWithPath [path=file:/Users/asinha/data/multilevel/parquet/1996/Q4/orders_96_q4.parquet]], selectionRoot=file:/Users/asinha/data/multilevel/parquet, numFiles=12, usedMetadataFile=false, columns=[`dir0`, `o_clerk`]]])

The log indicates no partition filters were found:

...
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for partition pruning.Total pruning elapsed time: 0 ms

A preliminary analysis indicates that since the Coalesce gets converted to a CASE(IS NOT NULL....) expression, the filter analysis does not correctly process the full expression tree. At one point in FindPartitionConditions.analyzeCall() I saw the operandStack had 3 elements in it: [NO_PUSH, NO_PUSH, PUSH] which seemed strange since I would expect even number of elements.