Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-1942 Improve off-heap memory usage tracking
  3. DRILL-3616

Memory leak in a cleanup code after canceling queries with window functions spilling to disk

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Execution - Flow
    • Labels:
      None
    • Environment:

      private-locking-allocator-branch

      Description

      Bunch of concurrent queries with window functions were cancelled.
      Got an error in drillbit.log that might indicate that we have a memory leak in in cleanup code after cancellation.

      Assigning to myself for creation of a reproducible test case.

      2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from CANCELLATION_REQUESTED --> FAILED
      2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED --> FAILED
      2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED --> FINISHED
      2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Unaccounted for outstanding allocation (902492)
      
      Fragment 0:0
      
      [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Unaccounted for outstanding allocation (902492)
      
      Fragment 0:0
      
      [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
      Caused by: java.lang.RuntimeException: Exception while closing
              at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              ... 5 common frames omitted
      Caused by: java.lang.IllegalStateException: Unaccounted for outstanding allocation (902492)
              at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
              ... 10 common frames omitted
      2015-08-05 22:43:56,477 [2a3d6e54-f2c1-4682-9121-73c9110e3dd7:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: ChannelClosedException: Channel closed /10.10.88.133:31010 <--> /10.10.88.133:45492.
      
      Fragment 0:0
      
      [Error Id: 16ee1618-f062-4a7b-af14-32a60e14d78f on atsqa4-133.qa.lab:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ChannelClosedException: Channel closed /10.10.88.133:31010 <--> /10.10.88.133:45492.
      

        Attachments

        1. DRILL-3616.1.patch.txt
          2 kB
          Deneche A. Hakim

          Issue Links

            Activity

              People

              • Assignee:
                vicky Victoria Markman
                Reporter:
                vicky Victoria Markman
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: