[IGNITE-7526] SQL: Introduce memory region for reducer merge results with disk offload

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: sql
• Labels: h2-limitation sql-stability
• Assignee: Unassigned
• Reporter: Vladimir Ozerov
• Created: 25/Jan/18 08:28
• Updated: 01/Oct/19 03:09
• Resolved: 06/May/19 12:10

Description

Currently all results received from map nodes are stored inside reducer's heap memory. What is worse, in case of complex queries, such as having sorts or groupings, need to collect all results from mappers first before final processing could be applied. In case of big results set (or intermediate results) this could easily lead to OOME on reducer.

To mitigate this we should introduce special memory area where intermediate results could be stored. All final processing should be stored in the same area as well. This area should be of limited size and should be able to offload results to disk in case of overflow.

We could start with our B+Tree and free list and store results in some K-V form.

Issue Links

• is related to: IGNITE-9480 SQL: Introduce H2 LocalResult factory
• is related to: IGNITE-9616 SQL: Introduce H2 factory for holder of aggregate result
• is related to: IGNITE-10598 H2 ConnectionManager drop ThreadLocal logic
• links to: GitHub Pull Request #4238
• links to: GitHub Pull Request #4238
• links to: Tests results