Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6252

Foreman node is going down when the non foreman node is stopped

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Two drillbits are running. I'm running a join query over parquet and tried to stop the non-foreman node using drillbit.sh stop. The query fails with "Error: DATA_READ ERROR: Exception occurred while reading from disk". The non-foreman node goes down. The foreman node also goes down. When I looked at the drillbit.log of both foreman and non-foreman I found that there is memory leak  "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 1000000/6291456/6832128/10000000000 (res/actual/peak/limit)\n". Following are the stack traces for memory leaks 

       
      [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
      
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
      
      Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)
      
       
      
       
      
      Fragment 2:1 
      
      [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
      
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
      
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
      
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
      
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
      
              at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
      
      Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
      
      Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)
      

       

       

      Ping me for the logs and more information.

       

        Attachments

        1. foreman_drillbit.log
          3.03 MB
          Venkata Jyothsna Donapati
        2. nonforeman_drillbit.log
          1.19 MB
          Venkata Jyothsna Donapati

          Issue Links

            Activity

              People

              • Assignee:
                vrozov Vlad Rozov
                Reporter:
                vdonapati Venkata Jyothsna Donapati
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: