[DRILL-6252] Foreman node is going down when the non foreman node is stopped

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Assignee: Vlad Rozov
• Reporter: Venkata Jyothsna Donapati
• Created: 15/Mar/18 18:57
• Updated: 13/Jun/18 00:18
• Resolved: 18/Apr/18 18:32

Description

Two drillbits are running. I'm running a join query over parquet and tried to stop the non-foreman node using drillbit.sh stop. The query fails with "Error: DATA_READ ERROR: Exception occurred while reading from disk". The non-foreman node goes down. The foreman node also goes down. When I looked at the drillbit.log of both foreman and non-foreman I found that there is memory leak  "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 1000000/6291456/6832128/10000000000 (res/actual/peak/limit)\n". Following are the stack traces for memory leaks 

 
[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)

 

 

Ping me for the logs and more information.

 

Issue Links

• blocks: DRILL-6039 drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit
• is caused by: DRILL-6295 PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation
• is duplicated by: DRILL-6039 drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit