Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20266

New Sources prevent JVM shutdown when running a job

    XMLWordPrintableJSON

    Details

      Description

      When trying out the new FileSource I noticed that the jobs which I started from my IDE won't properly terminate. To be more precise the spawned JVM for the jobs wouldn't properly terminate. I cannot really tell what the FileSource does differently, but when not using this source, the JVM terminates properly.

      The stack trace of the hanging JVM is

      2020-11-20 18:20:02
      Full thread dump OpenJDK 64-Bit Server VM (11.0.2+9 mixed mode):
      
      Threads class SMR info:
      _java_thread_list=0x00007fb5bc15f1b0, length=19, elements={
      0x00007fb60d807000, 0x00007fb60d80c000, 0x00007fb60d80f000, 0x00007fb60d809000,
      0x00007fb60d81a000, 0x00007fb61f00b000, 0x00007fb63d80e000, 0x00007fb61d826800,
      0x00007fb61d829800, 0x00007fb61e800000, 0x00007fb63d95d800, 0x00007fb63e2f8800,
      0x00007fb5ba37a800, 0x00007fb5afe1a800, 0x00007fb61dff6800, 0x00007fb63da49800,
      0x00007fb63e8d0800, 0x00007fb5be001000, 0x00007fb5bb8a4000
      }
      
      "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=10.05ms elapsed=86.35s tid=0x00007fb60d807000 nid=0x4b03 waiting on condition  [0x00007000036e9000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.2/Native Method)
      	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.2/Reference.java:241)
      	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.2/Reference.java:213)
      
      "Finalizer" #3 daemon prio=8 os_prio=31 cpu=0.90ms elapsed=86.35s tid=0x00007fb60d80c000 nid=0x3803 in Object.wait()  [0x00007000037ec000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(java.base@11.0.2/Native Method)
      	- waiting on <0x0000000600204780> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.2/ReferenceQueue.java:155)
      	- waiting to re-lock in wait() <0x0000000600204780> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.2/ReferenceQueue.java:176)
      	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.2/Finalizer.java:170)
      
      "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=0.31ms elapsed=86.34s tid=0x00007fb60d80f000 nid=0x4203 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 cpu=2479.36ms elapsed=86.34s tid=0x00007fb60d809000 nid=0x3f03 waiting on condition  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
         No compile task
      
      "C1 CompilerThread0" #8 daemon prio=9 os_prio=31 cpu=1412.88ms elapsed=86.34s tid=0x00007fb60d81a000 nid=0x3d03 waiting on condition  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
         No compile task
      
      "Sweeper thread" #9 daemon prio=9 os_prio=31 cpu=42.82ms elapsed=86.34s tid=0x00007fb61f00b000 nid=0xa803 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Common-Cleaner" #10 daemon prio=8 os_prio=31 cpu=3.25ms elapsed=86.29s tid=0x00007fb63d80e000 nid=0x5703 in Object.wait()  [0x0000700003cfb000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(java.base@11.0.2/Native Method)
      	- waiting on <0x0000000600205aa0> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.2/ReferenceQueue.java:155)
      	- waiting to re-lock in wait() <0x0000000600205aa0> (a java.lang.ref.ReferenceQueue$Lock)
      	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.2/CleanerImpl.java:148)
      	at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
      	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.2/InnocuousThread.java:134)
      
      "JDWP Transport Listener: dt_socket" #11 daemon prio=10 os_prio=31 cpu=43.46ms elapsed=86.27s tid=0x00007fb61d826800 nid=0xa603 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "JDWP Event Helper Thread" #12 daemon prio=10 os_prio=31 cpu=220.06ms elapsed=86.27s tid=0x00007fb61d829800 nid=0x5e03 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "JDWP Command Reader" #13 daemon prio=10 os_prio=31 cpu=27.26ms elapsed=86.27s tid=0x00007fb61e800000 nid=0x6103 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Service Thread" #14 daemon prio=9 os_prio=31 cpu=0.06ms elapsed=86.19s tid=0x00007fb63d95d800 nid=0xa203 runnable  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "ForkJoinPool.commonPool-worker-19" #25 daemon prio=1 os_prio=31 cpu=2.00ms elapsed=84.76s tid=0x00007fb63e2f8800 nid=0x8003 waiting on condition  [0x000070000584c000]
         java.lang.Thread.State: WAITING (parking)
      	at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
      	- parking to wait for  <0x0000000600a81188> (a java.util.concurrent.ForkJoinPool)
      	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.2/LockSupport.java:194)
      	at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.2/ForkJoinPool.java:1628)
      	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.2/ForkJoinWorkerThread.java:177)
      
      "ForkJoinPool.commonPool-worker-23" #58 daemon prio=5 os_prio=31 cpu=0.19ms elapsed=84.12s tid=0x00007fb5ba37a800 nid=0xcc03 waiting on condition  [0x0000700007197000]
         java.lang.Thread.State: WAITING (parking)
      	at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
      	- parking to wait for  <0x0000000600a81188> (a java.util.concurrent.ForkJoinPool)
      	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.2/LockSupport.java:194)
      	at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.2/ForkJoinPool.java:1628)
      	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.2/ForkJoinWorkerThread.java:177)
      
      "ForkJoinPool.commonPool-worker-9" #59 daemon prio=5 os_prio=31 cpu=0.17ms elapsed=84.07s tid=0x00007fb5afe1a800 nid=0xcf03 waiting on condition  [0x000070000729a000]
         java.lang.Thread.State: WAITING (parking)
      	at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
      	- parking to wait for  <0x0000000600a81188> (a java.util.concurrent.ForkJoinPool)
      	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.2/LockSupport.java:194)
      	at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.2/ForkJoinPool.java:1628)
      	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.2/ForkJoinWorkerThread.java:177)
      
      "ForkJoinPool.commonPool-worker-27" #63 daemon prio=5 os_prio=31 cpu=0.72ms elapsed=83.97s tid=0x00007fb61dff6800 nid=0xd503 waiting on condition  [0x00007000076a6000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
      	- parking to wait for  <0x0000000600a81188> (a java.util.concurrent.ForkJoinPool)
      	at java.util.concurrent.locks.LockSupport.parkUntil(java.base@11.0.2/LockSupport.java:275)
      	at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.2/ForkJoinPool.java:1619)
      	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.2/ForkJoinWorkerThread.java:177)
      
      "Cleaner-0" #185 daemon prio=8 os_prio=31 cpu=10.55ms elapsed=83.39s tid=0x00007fb63da49800 nid=0x1570b in Object.wait()  [0x00007000093fd000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(java.base@11.0.2/Native Method)
      	- waiting on <0x0000000600c5bd78> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.2/ReferenceQueue.java:155)
      	- waiting to re-lock in wait() <0x0000000600c5bd78> (a java.lang.ref.ReferenceQueue$Lock)
      	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.2/CleanerImpl.java:148)
      	at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
      	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.2/InnocuousThread.java:134)
      
      "ComponentClosingUtil" #232 prio=1 os_prio=31 cpu=0.39ms elapsed=81.78s tid=0x00007fb63e8d0800 nid=0x13107 waiting on condition  [0x0000700004f31000]
         java.lang.Thread.State: WAITING (parking)
      	at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
      	- parking to wait for  <0x00000006008fb930> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.2/LockSupport.java:194)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.2/AbstractQueuedSynchronizer.java:2081)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.2/ScheduledThreadPoolExecutor.java:1170)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.2/ScheduledThreadPoolExecutor.java:899)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.2/ThreadPoolExecutor.java:1054)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.2/ThreadPoolExecutor.java:1114)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.2/ThreadPoolExecutor.java:628)
      	at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
      
      "DestroyJavaVM" #282 prio=5 os_prio=31 cpu=2125.94ms elapsed=81.19s tid=0x00007fb5be001000 nid=0x1d03 waiting on condition  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Attach Listener" #291 daemon prio=9 os_prio=31 cpu=0.57ms elapsed=0.09s tid=0x00007fb5bb8a4000 nid=0x14207 waiting on condition  [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "VM Thread" os_prio=31 cpu=74.59ms elapsed=86.36s tid=0x00007fb63f807000 nid=0x3603 runnable
      
      "GC Thread#0" os_prio=31 cpu=51.88ms elapsed=86.38s tid=0x00007fb62d803800 nid=0x2e03 runnable
      
      "GC Thread#1" os_prio=31 cpu=50.32ms elapsed=85.97s tid=0x00007fb61d836800 nid=0x9e03 runnable
      
      "GC Thread#2" os_prio=31 cpu=45.79ms elapsed=85.97s tid=0x00007fb61d8b8000 nid=0x9d03 runnable
      
      "GC Thread#3" os_prio=31 cpu=46.08ms elapsed=85.97s tid=0x00007fb61d83a800 nid=0x9c03 runnable
      
      "GC Thread#4" os_prio=31 cpu=48.88ms elapsed=85.97s tid=0x00007fb61f01c000 nid=0x9a03 runnable
      
      "GC Thread#5" os_prio=31 cpu=44.32ms elapsed=85.97s tid=0x00007fb63d8e5000 nid=0x9803 runnable
      
      "GC Thread#6" os_prio=31 cpu=49.81ms elapsed=85.97s tid=0x00007fb63d8e6000 nid=0x9703 runnable
      
      "GC Thread#7" os_prio=31 cpu=51.55ms elapsed=85.97s tid=0x00007fb61d83b800 nid=0x6503 runnable
      
      "GC Thread#8" os_prio=31 cpu=46.05ms elapsed=85.97s tid=0x00007fb62d928000 nid=0x9403 runnable
      
      "GC Thread#9" os_prio=31 cpu=50.43ms elapsed=85.97s tid=0x00007fb62d929000 nid=0x9203 runnable
      
      "G1 Main Marker" os_prio=31 cpu=0.65ms elapsed=86.37s tid=0x00007fb62d83b000 nid=0x2f03 runnable
      
      "G1 Conc#0" os_prio=31 cpu=11.46ms elapsed=86.37s tid=0x00007fb62d83b800 nid=0x3203 runnable
      
      "G1 Conc#1" os_prio=31 cpu=11.43ms elapsed=85.39s tid=0x00007fb63da4e800 nid=0x6703 runnable
      
      "G1 Conc#2" os_prio=31 cpu=10.03ms elapsed=85.39s tid=0x00007fb63da4f800 nid=0x6803 runnable
      
      "G1 Refine#0" os_prio=31 cpu=8.22ms elapsed=86.37s tid=0x00007fb63e102000 nid=0x4e03 runnable
      
      "G1 Refine#1" os_prio=31 cpu=0.06ms elapsed=81.10s tid=0x00007fb5aeff4000 nid=0x1d50f runnable
      
      "G1 Young RemSet Sampling" os_prio=31 cpu=12.94ms elapsed=86.37s tid=0x00007fb63e102800 nid=0x3403 runnable
      "VM Periodic Task Thread" os_prio=31 cpu=57.93ms elapsed=86.19s tid=0x00007fb61e833800 nid=0xa003 waiting on condition
      
      JNI global refs: 55, weak refs: 15453
      

      My environment is and I used the openjdk version "1.8.0_262" to run the job:

      IntelliJ IDEA 2020.2.3 (Community Edition)
      Build #IC-202.7660.26, built on October 6, 2020
      Runtime version: 11.0.8+10-b944.34 x86_64
      VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
      macOS 10.14.6
      GC: ParNew, ConcurrentMarkSweep
      Memory: 1979M
      Cores: 12
      Non-Bundled Plugins: CheckStyle-IDEA, com.jetbrains.performancePlugin, org.jetbrains.kotlin, org.intellij.scala

      cc Stephan Ewen

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sewen Stephan Ewen
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: