[ASTERIXDB-2252] Improve scan efficiency of LSM components

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: STO - Storage
• Labels: triaged
• Assignee: Chen Luo
• Reporter: Chen Luo
• Created: 18/Jan/18 01:27
• Updated: 14/Mar/18 23:21

Description

The current (full) scan on LSM components is not very efficient, especially on hard disks, in two aspects:

1. We often need to use a priority queue to merge results from multiple components. However, we only read a page at a time, which incurs a lot of random I/O overhead on hard disks.
2. Full scan can often fill up (and clean up) the buffer cache. This problem is especially notable when we do merge. After a merge operation, the buffer cache would be filled up pages of old components, which would not be accessed by future queries.