[IGNITE-19412] Potential memory leak in MailboxRegistryImpl under intensive SQL load

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-beta1
• Fix Version/s: 3.0
• Component/s: sql
• Labels: ignite-3 sql
• Assignee: Konstantin Orlov
• Reporter: Andrey Khitrin
• Created: 03/May/23 12:53
• Updated: 04/Sep/24 16:10
• Resolved: 11/May/23 13:50

Description

I tried to run a SQL benchmark (slightly modified Benchbase TPC-C scenario) against Ignite3 (commit hash b48ddcba7cd2bd3b9a053ae131c25b44a0400e27).

Ignite3 was running with -Xmx6G -Xms6G memory limit. After some time it has reached memory limits and failed due to OutOfMemoryError. A heap dump was generated on failure. I can't attach this dump due to its size, but I've opened it in MAT and created a report (see zip file attached).

The report says that most of the retained heap (almost 5G) is occupied by a single instance of o.a.i.internal.sql.engine.exec.MailboxRegistryImpl class (see image attached).

Hypothesis: on an intense load, MailboxRegistryImpl creates a lot of lambda expressions. This happens faster than they're being utilized. With limited memory size, it could lead to OOM at some moment.

Benchmark is quite trivial by mechanics:

1. Create a table structure (see DDL file attached)
2. Fill tables with randomized data (several tens of thousands rows is enough)
3. Run select queries in a loop (actually, it wouldn't happen; OOM was raised on data insertion).

Issue Links

• is cloned by: IGNITE-19419 Internal queue overload under intensive SQL load
• is related to: IGNITE-19247 BatchUpdateException: Replication is timed out" upon inserting rows in batches via JDBC
• links to: GitHub Pull Request #2043