Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
v2.0.0
-
None
Description
Question
when a column has null value，and using it as a filter column when querying, and the filter value is not exist in the table,this may cause incorrect result
An Example
Table A
the table A has three rows,city column of one row has null value
| day | ... | city | price |
|---|---|---|---|
| 20180101 | null | 10 | |
| 20180101 | beijing | 20 | |
| 20180101 | shanghai | 10 |
Query SQL
select day,sum(price) from a where city <> 'abc' group by day
Correct Result
exclude the row contains null city value
| day | col |
|---|---|
| 20180101 | 30 |
InCorrect Result
resullt 0 rows
this happens in our production environment,the kylin version is 2.0.0
Analysis process
1,city column dosen't have a value,so the CompareTupleFilter will turn into ConstantTupleFilter(see GTUtil.java)
2,if dimensions in the sql dosen't match all the columns using in group by,the bytesComparator used in hbase aggregation map will only compare the columns using in group by
3,when GTAggregateScanner constructs key of aggBufMap,the key may contains null value,because the comparator of aggBufMap only compares group by columns,so the tuple share same group by columns may also share the same keys which contains null value;This may cause kylin server receives tuples contains null value;
4,when the code which dynamically generated by calcilte deals tuples using filter,it first judges whether the column is null.Because filter column in the tuple contains null value,so it always return false, no tuples will return.
Solution
when the filter column value is a invalid means not in the table,turn the CompareTupleFiter into IS_NOT_NULL filter,instead of ConstantTupleFilter.TURE
Now I have test the feature in our production environment ;
test in “mvn test” had passed，but not test in sandbox
Attachments
Attachments
Issue Links
- links to