[BEAM-7412] portable spark: thread/memory leak in local mode

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.14.0
• Component/s: runner-spark
• Labels: portability-spark
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 23/May/19 22:36
• Updated: 16/May/20 13:50
• Resolved: 04/Jun/19 20:35

Description

When running validatesPortableRunnerBatch test on local mode, the portable Spark runner creates ~18k threads before becoming unable to create any more threads, at which point it crashes. This does not happen on standalone cluster mode, where ephemeral executors (independent JVMs) are used and then shut down, preventing whatever leakage is occurring from becoming too much of a problem.

Sample stack trace:

"pool-3965-thread-3" #16059 daemon prio=5 os_prio=0 tid=0x00007f9f85a81000 nid=0x32a0 waiting on condition [0x00007f9f8b9f1000]
{{ java.lang.Thread.State: TIMED_WAITING (parking)}}
{{ at (C/C++) 0x00007fa2f32c2dae (Unknown Source)}}
{{ at (C/C++) 0x00007fa2f2851351 (Unknown Source)}}
{{ at sun.misc.Unsafe.park(Native Method)}}
{{ - parking to wait for <0x0000000727bda848> (a java.util.concurrent.SynchronousQueue$TransferStack)}}
{{ at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)}}
{{ at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)}}
{{ at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)}}
{{ at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)}}
{{ at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)}}
{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)}}
{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}

Issue Links

• blocks: BEAM-7175 Add Spark Portable ValidatesRunner Batch postcommit test
• links to: GitHub Pull Request #8722