[DRILL-8479] Merge Join Memory Leak Depleting Incoming Batches Throw Exception

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.21.1
• Fix Version/s: 1.21.2
• Component/s: Execution - Relational Operators
• Assignee: Unassigned
• Reporter: shihuafeng
• Reviewer: Paul Rogers
• Created: 24/Jan/24 05:42
• Updated: 15/Apr/24 07:13
• Resolved: 15/Apr/24 07:13

Description

Describe the bug
megerjoin  leak when RecordIterator allocate memory exception with OutOfMemoryException{}
Steps to reproduce the behavior:

1. prepare data for tpch 1s
2. set direct memory 5g
3.  set planner.enable_hashjoin =false  to  ensure use mergejoin operator。
4.  set drill.memory.debug.allocator =true (Check for memory leaks )
5. 20 concurrent for tpch sql8
6. when it had OutOfMemoryException or null EXCEPTION , stopped all sql.
7. finding memory leak

Expected behavior

      when all  sql sop , we should find direct memory is 0 AND  could not find leak log like following.

Allocator(op:2:0:11:MergeJoinPOP) 1000000/73728/4874240/10000000000 (res/actual/peak/limit)

Error detail, log output or screenshots

Unable to allocate buffer of size XX (rounded from XX) due to memory limit (). Current allocation: xx

0001-mergejoin-leak.patch

sql 

// code placeholder
select o_year, sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as all_nations group by o_year order by o_year