Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7412

portable spark: thread/memory leak in local mode

    XMLWordPrintableJSON

    Details

      Description

      When running validatesPortableRunnerBatch test on local mode, the portable Spark runner creates ~18k threads before becoming unable to create any more threads, at which point it crashes. This does not happen on standalone cluster mode, where ephemeral executors (independent JVMs) are used and then shut down, preventing whatever leakage is occurring from becoming too much of a problem.

      Sample stack trace:

      "pool-3965-thread-3" #16059 daemon prio=5 os_prio=0 tid=0x00007f9f85a81000 nid=0x32a0 waiting on condition [0x00007f9f8b9f1000]
      {{ java.lang.Thread.State: TIMED_WAITING (parking)}}
      {{ at (C/C++) 0x00007fa2f32c2dae (Unknown Source)}}
      {{ at (C/C++) 0x00007fa2f2851351 (Unknown Source)}}
      {{ at sun.misc.Unsafe.park(Native Method)}}
      {{ - parking to wait for <0x0000000727bda848> (a java.util.concurrent.SynchronousQueue$TransferStack)}}
      {{ at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)}}
      {{ at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)}}
      {{ at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)}}
      {{ at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      {{ at java.lang.Thread.run(Thread.java:748)}}

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ibzib Kyle Weaver
                Reporter:
                ibzib Kyle Weaver
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h