[PHOENIX-6751] Force using range scan vs skip scan when using the IN operator and large number of RVC elements - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 4.15.0, 5.1.1, 4.16.0, 5.2.0
Fix Version/s: 5.2.0, 5.1.3
Component/s: None
Labels:
None

Release Note:
Adds a new config parameter, phoenix.max.inList.skipScan.size, which controls the size of an IN clause before it will be automatically converted from a skip scan to a range scan.

Description

SQL queries using the IN operator using PKs of different SortOrder were failing during the WHERE clause compilation phase and causing OOM issues on the servers when a large number (~50k) of RVC elements were used in the IN operator.

SQL queries were failing specifically during the skip scan filter generation. The skip scan filter is generated using a list of point key ranges. ScanRanges.create

The following getPointKeys code uses the KeyRange sets to create a new list of point-keys. When there are a large number of RVC elements the above