[DRILL-6132] HashPartitionSender leaks memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.14.0
• Component/s: Functions - Drill
• Assignee: Timothy Farkas
• Reporter: Chun Chang
• Created: 02/Feb/18 19:35
• Updated: 11/Apr/18 21:58
• Resolved: 11/Apr/18 21:58

Description

Enable assertion (-ea), I noticed HashPartitionSender leaks memory if aggregation query fails due to OOM.

message: "SYSTEM ERROR: IllegalStateException: Allocator[op:2:1:0:HashPartitionSender] closed with outstanding buffers allocated (1).\nAllocator(op:2:1:0:HashPartitionSender) 1000000/8192/3300352/10000000000 (res/actual/peak/limit)\n child allocators: 0\n ledgers: 1\n ledger[703835] allocator: op:2:1:0:HashPartitionSender), isOwning: true, size: 8192, references: 1, life: 6329151004063490..0, allocatorManager: [688058, life: 6329151004058252..0] holds 1 buffers. \n DrillBuf[777552], udle: [688059 0..8192]: , \n reservations: 0\n\n\nFragment 2:1\n\n[Error Id: c7cc9d37-8881-4db1-8123-2651628c4081 on 10.10.30.168:31010]\n\n (java.lang.IllegalStateException) Allocator[op:2:1:0:HashPartitionSender] closed with outstanding buffers allocated (1).\nAllocator(op:2:1:0:HashPartitionSender) 1000000/8192/3300352/10000000000 (res/actual/peak/limit)\n child allocators: 0\n ledgers: 1\n ledger[703835] allocator: op:2:1:0:HashPartitionSender), isOwning: true, size: 8192, references: 1, life: 6329151004063490..0, allocatorManager: [688058, life: 6329151004058252..0] holds 1 buffers. \n DrillBuf[777552], udle: [688059 0..8192]: , \n reservations: 0\n\n org.apache.drill.exec.memory.BaseAllocator.close():504\n org.apache.drill.exec.ops.BaseOperatorContext.close():157\n org.apache.drill.exec.ops.OperatorContextImpl.close():79\n org.apache.drill.exec.ops.FragmentContext.suppressingClose():429\n org.apache.drill.exec.ops.FragmentContext.close():418\n org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():324\n org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():155\n org.apache.drill.exec.work.fragment.FragmentExecutor.run():267\n org.apache.drill.common.SelfCleaningRunnable.run():38\n java.util.concurrent.ThreadPoolExecutor.runWorker():1149\n java.util.concurrent.ThreadPoolExecutor$Worker.run():624\n java.lang.Thread.run():748\n"
exception {
exception_class: "java.lang.IllegalStateException"
message: "Allocator[op:2:1:0:HashPartitionSender] closed with outstanding buffers allocated (1).\nAllocator(op:2:1:0:HashPartitionSender) 1000000/8192/3300352/10000000000 (res/actual/peak/limit)\n child allocators: 0\n ledgers: 1\n ledger[703835] allocator: op:2:1:0:HashPartitionSender), isOwning: true, size: 8192, references: 1, life: 6329151004063490..0, allocatorManager: [688058, life: 6329151004058252..0] holds 1 buffers. \n DrillBuf[777552], udle: [688059 0..8192]: , \n reservations: 0\n"
stack_trace {
class_name: "org.apache.drill.exec.memory.BaseAllocator"
file_name: "BaseAllocator.java"
line_number: 504
method_name: "close"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.ops.BaseOperatorContext"
file_name: "BaseOperatorContext.java"
line_number: 157
method_name: "close"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.ops.OperatorContextImpl"
file_name: "OperatorContextImpl.java"
line_number: 79
method_name: "close"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.ops.FragmentContext"
file_name: "FragmentContext.java"
line_number: 429
method_name: "suppressingClose"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.ops.FragmentContext"
file_name: "FragmentContext.java"
line_number: 418
method_name: "close"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor"
file_name: "FragmentExecutor.java"
line_number: 324
method_name: "closeOutResources"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor"
file_name: "FragmentExecutor.java"
line_number: 155
method_name: "cleanup"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor"
file_name: "FragmentExecutor.java"
line_number: 267
method_name: "run"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.common.SelfCleaningRunnable"
file_name: "SelfCleaningRunnable.java"
line_number: 38
method_name: "run"
is_native_method: false
}
stack_trace {
class_name: "..."
line_number: 0
method_name: "..."
is_native_method: false
}
}
}

Issue Links

• relates to: DRILL-6125 PartitionSenderRootExec can leak memory because close method is not synchronized