[FLINK-21004] test_process_mode_boot.py test hangs

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0
• Component/s: API / Python
• Labels: auto-deprioritized-major test-stability
• Assignee: Huang Xingbo
• Reporter: Huang Xingbo
• Created: 18/Jan/21 06:17
• Updated: 24/Aug/21 07:23
• Resolved: 24/Aug/21 07:23

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12159&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=4fad9527-b9a5-5015-1b70-8356e5c91490

2021-01-18T00:55:48.3027307Z pyflink/fn_execution/tests/test_process_mode_boot.py ==============================================================================
2021-01-18T00:55:48.3028351Z Process produced no output for 900 seconds.
2021-01-18T00:55:48.3033084Z ==============================================================================
2021-01-18T00:55:48.3033633Z ==============================================================================
2021-01-18T00:55:48.3034073Z The following Java processes are running (JPS)
2021-01-18T00:55:48.3037261Z ==============================================================================
2021-01-18T00:55:48.3180991Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2021-01-18T00:55:48.4930672Z 18493 Jps
2021-01-18T00:55:48.4931189Z 12477 PythonGatewayServer
2021-01-18T00:55:48.4979543Z ==============================================================================
2021-01-18T00:55:48.4984759Z Printing stack trace of Java process 18493
2021-01-18T00:55:48.4987182Z ==============================================================================
2021-01-18T00:55:48.5025804Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2021-01-18T00:55:48.5943552Z 18493: No such process
2021-01-18T00:55:48.6089460Z ==============================================================================
2021-01-18T00:55:48.6089977Z Printing stack trace of Java process 12477
2021-01-18T00:55:48.6094322Z ==============================================================================
2021-01-18T00:55:48.6140780Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2021-01-18T00:55:48.9394259Z 2021-01-18 00:55:48
2021-01-18T00:55:48.9401959Z Full thread dump OpenJDK 64-Bit Server VM (25.275-b01 mixed mode):
2021-01-18T00:55:48.9402608Z 
2021-01-18T00:55:48.9403205Z "Attach Listener" #3327 daemon prio=9 os_prio=0 tid=0x00007f4d9c02c800 nid=0x4864 waiting on condition [0x0000000000000000]
2021-01-18T00:55:48.9403817Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9404137Z 
2021-01-18T00:55:48.9404634Z "process reaper" #2273 daemon prio=10 os_prio=0 tid=0x00007f4db809c000 nid=0x3ff9 runnable [0x00007f4d7b070000]
2021-01-18T00:55:48.9405191Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9405785Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9425226Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9431982Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9432532Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9439963Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9444803Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9445051Z 
2021-01-18T00:55:48.9452857Z "process reaper" #2033 daemon prio=10 os_prio=0 tid=0x00007f4db8086800 nid=0x3e2a runnable [0x00007f4d7b0e2000]
2021-01-18T00:55:48.9453377Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9453743Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9454188Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9454645Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9455136Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9455669Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9456119Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9456355Z 
2021-01-18T00:55:48.9456708Z "process reaper" #1923 daemon prio=10 os_prio=0 tid=0x00007f4d9c005800 nid=0x3cef runnable [0x00007f4d7808b000]
2021-01-18T00:55:48.9457148Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9457484Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9457915Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9458361Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9458814Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9459364Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9460033Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9460247Z 
2021-01-18T00:55:48.9460619Z "process reaper" #1922 daemon prio=10 os_prio=0 tid=0x00007f4d8c05a800 nid=0x3ceb runnable [0x00007f4d780c4000]
2021-01-18T00:55:48.9461042Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9461399Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9461833Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9462260Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9462738Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9463281Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9463707Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9464046Z 
2021-01-18T00:55:48.9464396Z "process reaper" #1921 daemon prio=10 os_prio=0 tid=0x00007f4d9834a800 nid=0x3ce1 runnable [0x00007f4d7b31e000]
2021-01-18T00:55:48.9464850Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9465206Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9465619Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9466228Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9466708Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9467227Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9467678Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9467886Z 
2021-01-18T00:55:48.9468256Z "process reaper" #1920 daemon prio=10 os_prio=0 tid=0x00007f4db0040000 nid=0x3cd9 runnable [0x00007f4d7b88b000]
2021-01-18T00:55:48.9468703Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9469042Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9469479Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9469926Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9470378Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9470919Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9471343Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9471575Z 
2021-01-18T00:55:48.9471921Z "process reaper" #1573 daemon prio=10 os_prio=0 tid=0x00007f4db8081000 nid=0x3a5c runnable [0x00007f4d7b8c4000]
2021-01-18T00:55:48.9472365Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9472723Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9473364Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9473796Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9474250Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9474747Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9475169Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9475371Z 
2021-01-18T00:55:48.9475716Z "process reaper" #1485 daemon prio=10 os_prio=0 tid=0x0000563e82288800 nid=0x399f runnable [0x00007f4d7b8fd000]
2021-01-18T00:55:48.9476149Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9476482Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9476913Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9477356Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9477811Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9478356Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9478929Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9479141Z 
2021-01-18T00:55:48.9479498Z "process reaper" #1484 daemon prio=10 os_prio=0 tid=0x00007f4db8080000 nid=0x399c runnable [0x00007f4da0047000]
2021-01-18T00:55:48.9479918Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9480268Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9480696Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9481121Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9481591Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9482125Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9482551Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9482775Z 
2021-01-18T00:55:48.9483186Z "process reaper" #1062 daemon prio=10 os_prio=0 tid=0x00007f4d800e1000 nid=0x3663 runnable [0x00007f4da0080000]
2021-01-18T00:55:48.9483626Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9483964Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9484389Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9484828Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9485281Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9485816Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9486254Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9486462Z 
2021-01-18T00:55:48.9486819Z "process reaper" #1061 daemon prio=10 os_prio=0 tid=0x0000563e82288000 nid=0x3662 runnable [0x00007f4da0e44000]
2021-01-18T00:55:48.9487238Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9487590Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9488017Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9488438Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9488910Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9489441Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9489867Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9490093Z 
2021-01-18T00:55:48.9491201Z "grpc-nio-worker-ELG-15-2" #768 daemon prio=5 os_prio=0 tid=0x0000563e82283000 nid=0x350f runnable [0x00007f4d78dc9000]
2021-01-18T00:55:48.9491655Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9492002Z 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
2021-01-18T00:55:48.9492404Z 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
2021-01-18T00:55:48.9492879Z 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
2021-01-18T00:55:48.9493358Z 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
2021-01-18T00:55:48.9494144Z 	- locked <0x00000000942a63f8> (a org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySet)
2021-01-18T00:55:48.9494859Z 	- locked <0x00000000942b93b8> (a java.util.Collections$UnmodifiableSet)
2021-01-18T00:55:48.9495471Z 	- locked <0x00000000942a62f0> (a sun.nio.ch.EPollSelectorImpl)
2021-01-18T00:55:48.9495909Z 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
2021-01-18T00:55:48.9496527Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
2021-01-18T00:55:48.9497250Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:824)
2021-01-18T00:55:48.9497862Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
2021-01-18T00:55:48.9498556Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
2021-01-18T00:55:48.9499352Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2021-01-18T00:55:48.9500040Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2021-01-18T00:55:48.9500585Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9500796Z 
2021-01-18T00:55:48.9501478Z "grpc-nio-worker-ELG-15-1" #766 daemon prio=5 os_prio=0 tid=0x0000563e82281000 nid=0x3509 runnable [0x00007f4d791cb000]
2021-01-18T00:55:48.9501930Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9502261Z 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
2021-01-18T00:55:48.9502677Z 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
2021-01-18T00:55:48.9503146Z 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
2021-01-18T00:55:48.9503681Z 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
2021-01-18T00:55:48.9504464Z 	- locked <0x00000000942cd5e8> (a org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySet)
2021-01-18T00:55:48.9505173Z 	- locked <0x00000000942d62f8> (a java.util.Collections$UnmodifiableSet)
2021-01-18T00:55:48.9505770Z 	- locked <0x00000000942cd4e0> (a sun.nio.ch.EPollSelectorImpl)
2021-01-18T00:55:48.9506180Z 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
2021-01-18T00:55:48.9506787Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
2021-01-18T00:55:48.9507486Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:824)
2021-01-18T00:55:48.9508111Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
2021-01-18T00:55:48.9508799Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
2021-01-18T00:55:48.9509503Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2021-01-18T00:55:48.9510206Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2021-01-18T00:55:48.9510741Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9510950Z 
2021-01-18T00:55:48.9511597Z "grpc-nio-boss-ELG-14-1" #764 daemon prio=5 os_prio=0 tid=0x00007f4d88db4800 nid=0x34f3 runnable [0x00007f4d7a2d8000]
2021-01-18T00:55:48.9512027Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9512376Z 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
2021-01-18T00:55:48.9512787Z 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
2021-01-18T00:55:48.9513398Z 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
2021-01-18T00:55:48.9513878Z 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
2021-01-18T00:55:48.9514672Z 	- locked <0x0000000094299a78> (a org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySet)
2021-01-18T00:55:48.9515364Z 	- locked <0x00000000942eb6b8> (a java.util.Collections$UnmodifiableSet)
2021-01-18T00:55:48.9515971Z 	- locked <0x00000000942999d0> (a sun.nio.ch.EPollSelectorImpl)
2021-01-18T00:55:48.9516380Z 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
2021-01-18T00:55:48.9516975Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
2021-01-18T00:55:48.9517690Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:824)
2021-01-18T00:55:48.9518312Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
2021-01-18T00:55:48.9518987Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
2021-01-18T00:55:48.9519797Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2021-01-18T00:55:48.9520495Z 	at org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2021-01-18T00:55:48.9521014Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9521241Z 
2021-01-18T00:55:48.9521588Z "process reaper" #152 daemon prio=10 os_prio=0 tid=0x00007f4da83e5000 nid=0x315b runnable [0x00007f4da00b9000]
2021-01-18T00:55:48.9522021Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9522378Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9522791Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9523239Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9523718Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9524315Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9524764Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9524974Z 
2021-01-18T00:55:48.9525335Z "process reaper" #150 daemon prio=10 os_prio=0 tid=0x00007f4da4192000 nid=0x3158 runnable [0x00007f4da0e03000]
2021-01-18T00:55:48.9526243Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9526609Z 	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
2021-01-18T00:55:48.9527039Z 	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
2021-01-18T00:55:48.9527482Z 	at java.lang.UNIXProcess$$Lambda$1358/33063055.run(Unknown Source)
2021-01-18T00:55:48.9527940Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-01-18T00:55:48.9528477Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9528929Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9529144Z 
2021-01-18T00:55:48.9529895Z "Thread-6" #18 daemon prio=5 os_prio=0 tid=0x00007f4d80008800 nid=0x30d3 runnable [0x00007f4da1d47000]
2021-01-18T00:55:48.9530305Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9530652Z 	at java.net.SocketInputStream.socketRead0(Native Method)
2021-01-18T00:55:48.9531077Z 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2021-01-18T00:55:48.9531535Z 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
2021-01-18T00:55:48.9531986Z 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
2021-01-18T00:55:48.9532421Z 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
2021-01-18T00:55:48.9532866Z 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
2021-01-18T00:55:48.9533298Z 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
2021-01-18T00:55:48.9536016Z 	- locked <0x0000000093104348> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9536468Z 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
2021-01-18T00:55:48.9536909Z 	at java.io.BufferedReader.fill(BufferedReader.java:161)
2021-01-18T00:55:48.9537331Z 	at java.io.BufferedReader.readLine(BufferedReader.java:324)
2021-01-18T00:55:48.9537944Z 	- locked <0x0000000093104348> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9538352Z 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
2021-01-18T00:55:48.9538844Z 	at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:230)
2021-01-18T00:55:48.9539314Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9539524Z 
2021-01-18T00:55:48.9540157Z "Thread-5" #17 daemon prio=5 os_prio=0 tid=0x00007f4d88015000 nid=0x30d1 waiting on condition [0x00007f4da1b45000]
2021-01-18T00:55:48.9540634Z    java.lang.Thread.State: TIMED_WAITING (parking)
2021-01-18T00:55:48.9540954Z 	at sun.misc.Unsafe.park(Native Method)
2021-01-18T00:55:48.9541655Z 	- parking to wait for  <0x0000000093163b70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2021-01-18T00:55:48.9542378Z 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
2021-01-18T00:55:48.9543289Z 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
2021-01-18T00:55:48.9543995Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
2021-01-18T00:55:48.9544660Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
2021-01-18T00:55:48.9545239Z 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
2021-01-18T00:55:48.9545775Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
2021-01-18T00:55:48.9546317Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9546741Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9547055Z 
2021-01-18T00:55:48.9547707Z "Thread-4" #15 daemon prio=5 os_prio=0 tid=0x00007f4d80006800 nid=0x30cf runnable [0x00007f4da1e48000]
2021-01-18T00:55:48.9548145Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9548491Z 	at java.net.SocketInputStream.socketRead0(Native Method)
2021-01-18T00:55:48.9548900Z 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2021-01-18T00:55:48.9549372Z 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
2021-01-18T00:55:48.9549826Z 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
2021-01-18T00:55:48.9550261Z 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
2021-01-18T00:55:48.9550703Z 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
2021-01-18T00:55:48.9551137Z 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
2021-01-18T00:55:48.9551733Z 	- locked <0x0000000093121b50> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9552148Z 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
2021-01-18T00:55:48.9552572Z 	at java.io.BufferedReader.fill(BufferedReader.java:161)
2021-01-18T00:55:48.9553781Z 	at java.io.BufferedReader.readLine(BufferedReader.java:324)
2021-01-18T00:55:48.9554412Z 	- locked <0x0000000093121b50> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9554803Z 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
2021-01-18T00:55:48.9555311Z 	at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:230)
2021-01-18T00:55:48.9555776Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9555984Z 
2021-01-18T00:55:48.9556582Z "Thread-3" #14 daemon prio=5 os_prio=0 tid=0x00007f4d88016000 nid=0x30cd runnable [0x00007f4da1c46000]
2021-01-18T00:55:48.9556998Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9557345Z 	at java.net.PlainSocketImpl.socketAccept(Native Method)
2021-01-18T00:55:48.9557791Z 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
2021-01-18T00:55:48.9558257Z 	at java.net.ServerSocket.implAccept(ServerSocket.java:560)
2021-01-18T00:55:48.9558698Z 	at java.net.ServerSocket.accept(ServerSocket.java:528)
2021-01-18T00:55:48.9559181Z 	at org.apache.flink.api.python.shaded.py4j.GatewayServer.run(GatewayServer.java:685)
2021-01-18T00:55:48.9559612Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9559838Z 
2021-01-18T00:55:48.9560502Z "Log4j2-TF-2-Scheduled-1" #9 daemon prio=5 os_prio=0 tid=0x00007f4dcc5bb000 nid=0x30c9 waiting on condition [0x00007f4da2149000]
2021-01-18T00:55:48.9561002Z    java.lang.Thread.State: TIMED_WAITING (parking)
2021-01-18T00:55:48.9561339Z 	at sun.misc.Unsafe.park(Native Method)
2021-01-18T00:55:48.9562023Z 	- parking to wait for  <0x0000000093114e18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2021-01-18T00:55:48.9562562Z 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
2021-01-18T00:55:48.9563167Z 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
2021-01-18T00:55:48.9563971Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
2021-01-18T00:55:48.9564639Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
2021-01-18T00:55:48.9565232Z 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
2021-01-18T00:55:48.9565747Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
2021-01-18T00:55:48.9566288Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-01-18T00:55:48.9566729Z 	at java.lang.Thread.run(Thread.java:748)
2021-01-18T00:55:48.9566938Z 
2021-01-18T00:55:48.9567594Z "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f4dcc0cf800 nid=0x30c7 runnable [0x0000000000000000]
2021-01-18T00:55:48.9568043Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9568257Z 
2021-01-18T00:55:48.9568741Z "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4dcc0ba800 nid=0x30c6 waiting on condition [0x0000000000000000]
2021-01-18T00:55:48.9569202Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9569396Z 
2021-01-18T00:55:48.9569778Z "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4dcc0b7800 nid=0x30c5 waiting on condition [0x0000000000000000]
2021-01-18T00:55:48.9570232Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9570423Z 
2021-01-18T00:55:48.9570782Z "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4dcc0b4800 nid=0x30c4 runnable [0x0000000000000000]
2021-01-18T00:55:48.9571192Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9571403Z 
2021-01-18T00:55:48.9571749Z "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4dcc084800 nid=0x30c3 in Object.wait() [0x00007f4db4277000]
2021-01-18T00:55:48.9572224Z    java.lang.Thread.State: WAITING (on object monitor)
2021-01-18T00:55:48.9572570Z 	at java.lang.Object.wait(Native Method)
2021-01-18T00:55:48.9572941Z 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
2021-01-18T00:55:48.9573705Z 	- locked <0x00000000930da6d0> (a java.lang.ref.ReferenceQueue$Lock)
2021-01-18T00:55:48.9574128Z 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
2021-01-18T00:55:48.9574595Z 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
2021-01-18T00:55:48.9574864Z 
2021-01-18T00:55:48.9575243Z "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4dcc080000 nid=0x30c2 in Object.wait() [0x00007f4db4378000]
2021-01-18T00:55:48.9575734Z    java.lang.Thread.State: WAITING (on object monitor)
2021-01-18T00:55:48.9576069Z 	at java.lang.Object.wait(Native Method)
2021-01-18T00:55:48.9576398Z 	at java.lang.Object.wait(Object.java:502)
2021-01-18T00:55:48.9576805Z 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
2021-01-18T00:55:48.9577485Z 	- locked <0x00000000931fc5d0> (a java.lang.ref.Reference$Lock)
2021-01-18T00:55:48.9577923Z 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
2021-01-18T00:55:48.9578204Z 
2021-01-18T00:55:48.9578538Z "main" #1 prio=5 os_prio=0 tid=0x00007f4dcc00c800 nid=0x30be runnable [0x00007f4dd459c000]
2021-01-18T00:55:48.9578946Z    java.lang.Thread.State: RUNNABLE
2021-01-18T00:55:48.9579272Z 	at java.net.SocketInputStream.socketRead0(Native Method)
2021-01-18T00:55:48.9579698Z 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2021-01-18T00:55:48.9580168Z 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
2021-01-18T00:55:48.9580600Z 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
2021-01-18T00:55:48.9581046Z 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
2021-01-18T00:55:48.9581488Z 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
2021-01-18T00:55:48.9581905Z 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
2021-01-18T00:55:48.9582512Z 	- locked <0x00000000931fe7e0> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9582913Z 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
2021-01-18T00:55:48.9583431Z 	at java.io.BufferedReader.fill(BufferedReader.java:161)
2021-01-18T00:55:48.9583862Z 	at java.io.BufferedReader.readLine(BufferedReader.java:324)
2021-01-18T00:55:48.9586297Z 	- locked <0x00000000931fe7e0> (a java.io.InputStreamReader)
2021-01-18T00:55:48.9586749Z 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
2021-01-18T00:55:48.9587305Z 	at org.apache.flink.api.python.shaded.py4j.CallbackConnection.readBlockingResponse(CallbackConnection.java:169)
2021-01-18T00:55:48.9587935Z 	at org.apache.flink.api.python.shaded.py4j.CallbackConnection.sendCommand(CallbackConnection.java:148)
2021-01-18T00:55:48.9588547Z 	at org.apache.flink.api.python.shaded.py4j.CallbackClient.sendCommand(CallbackClient.java:384)
2021-01-18T00:55:48.9589131Z 	at org.apache.flink.api.python.shaded.py4j.CallbackClient.sendCommand(CallbackClient.java:356)
2021-01-18T00:55:48.9589782Z 	at org.apache.flink.api.python.shaded.py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
2021-01-18T00:55:48.9590458Z 	at com.sun.proxy.$Proxy19.ping(Unknown Source)
2021-01-18T00:55:48.9590908Z 	at org.apache.flink.client.python.PythonGatewayServer.main(PythonGatewayServer.java:94)

Issue Links

• duplicates: FLINK-23493 python tests hang on Azure