[DRILL-4310] Memory leak in hash partition sender when query is cancelled

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.5.0
• Component/s: Execution - Flow
• Assignee: Unassigned
• Reporter: Victoria Markman
• Created: 26/Jan/16 02:23
• Updated: 27/Jun/17 06:36

Description

Query got cancelled (still investigating what caused cancellation).

Here is an excerpt from drillbit.log

2016-01-26 00:46:29,627 [29593ea8-88d2-612e-7c58-aa11652c4072:frag:2:2] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Allocator[op:2:2:0:HashPartitionSender] closed with outstanding buffers allocated (4).
Allocator(op:2:2:0:HashPartitionSender) 1000000/10240/2140160/10000000000 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 4
    ledger[10892635] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 4096, references: 1, life: 23697371310917183..0, allocatorManager: [7140397, life: 23697371310913697..0] holds 1 buffers.
        DrillBuf[13122380], udle: [7140398 0..4096]
    ledger[10892636] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 1024, references: 1, life: 23697371311045504..0, allocatorManager: [7140398, life: 23697371311041789..0] holds 1 buffers.
        DrillBuf[13122381], udle: [7140399 0..1024]
    ledger[10892634] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 4096, references: 1, life: 23697371310795164..0, allocatorManager: [7140396, life: 23697371310789988..0] holds 1 buffers.
        DrillBuf[13122379], udle: [7140397 0..4096]
    ledger[10892513] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 1024, references: 1, life: 23697371288488073..0, allocatorManager: [7140275, life: 23697371288484282..0] holds 1 buffers.
        DrillBuf[13122245], udle: [7140276 0..1024]
  reservations: 0

Fragment 2:2

[Error Id: 043c5d25-c4de-4a70-9cb1-d4987822ee3b on atsqa4-134.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Allocator[op:2:2:0:HashPartitionSender] closed with outstanding buffers allocated (4).
Allocator(op:2:2:0:HashPartitionSender) 1000000/10240/2140160/10000000000 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 4
    ledger[10892635] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 4096, references: 1, life: 23697371310917183..0, allocatorManager: [7140397, life: 23697371310913697..0] holds 1 buffers.
        DrillBuf[13122380], udle: [7140398 0..4096]
    ledger[10892636] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 1024, references: 1, life: 23697371311045504..0, allocatorManager: [7140398, life: 23697371311041789..0] holds 1 buffers.
        DrillBuf[13122381], udle: [7140399 0..1024]
    ledger[10892634] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 4096, references: 1, life: 23697371310795164..0, allocatorManager: [7140396, life: 23697371310789988..0] holds 1 buffers.
        DrillBuf[13122379], udle: [7140397 0..4096]
    ledger[10892513] allocator: op:2:2:0:HashPartitionSender), isOwning: true, size: 1024, references: 1, life: 23697371288488073..0, allocatorManager: [7140275, life: 23697371288484282..0] holds 1 buffers.
        DrillBuf[13122245], udle: [7140276 0..1024]
  reservations: 0

Reproduced twice by running: ./run.sh -s Advanced/tpcds/tpcds_sf100/original -g smoke -t 600 -n 10 -i 100 -m

Cluster configuration: vanilla, 48GB of memory, 4GB heap.

Attaching query profile and logs.