[KYLIN-2173] push down limit leads to wrong answer when filter is loosened

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v1.5.4.1
• Fix Version/s: v1.6.0
• Component/s: Storage - HBase
• Assignee: Dayue Gao
• Reporter: Dayue Gao
• Created: 09/Nov/16 08:18
• Updated: 26/Nov/16 08:12
• Resolved: 23/Nov/16 05:41

Description

To reproduce:

select
 test_kylin_fact.cal_dt
 ,sum(test_kylin_fact.price) as GMV
 FROM test_kylin_fact 
 left JOIN edw.test_cal_dt as test_cal_dt 
 ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt 
 where test_cal_dt.week_beg_dt in ('2012-01-01', '2012-01-20')
 group by test_kylin_fact.cal_dt 
 limit 12

Kylin returns 5 rows, expect 12 rows.

Root cause: filter condition may be loosened when we translate derived filter in DerivedFilterTranslator. If we push down limit, query server won't get enough valid records from storage. In the above example, 24 rows returned from storage, only 5 are valid.