[KYLIN-5007] queries with limit clause may fail when string dimension is encoded in integer type

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v3.0.2
• Fix Version/s: v3.1.3
• Component/s: Query Engine
• Assignee: Congling Xia
• Reporter: Congling Xia
• Created: 10/Jun/21 02:10
• Updated: 06/Jan/22 06:52
• Resolved: 09/Nov/21 09:22

Description

Hi, team.

Recently we encounter a problem that queries may fail if there is a LIMIT in the SQL. The SQL looks like:

select gid from some_table group by gid limit 100

The error message is like the following:

Not sorted! last: source_v1=null,...,gid=276,... fetched: source_v1=null,...,gid=100506,...

After searching the issues list, we find it is similar with KYLIN-2425, KYLIN-3089, and KYLIN-4942. We notice that these problems are not completely resolved.

It is an row-key encoding problem, the cube uses integer:4 to encode string column gid:

As kangkaisen mensioned in KYLIN-3089, comparator in SortMergedPartitionResultIterator is different from the one in SortedIteratorMergerWithLimit. SortedIteratorMergerWithLimit compares tuple of dimensions in their origin data type "string" rather than the encoded data type "integer" in rowkeys. In the exception message above, 276<100506 is false because they are compared in "string" type.

It may be resolved by skipping limit pushdown when column type and encoding type may produce different comparing results, but it may lead such queries to be slower.