[FLINK-26037] TaskManagerRunner may crash during shutdown sequence

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.15.0, 1.16.0, 1.17.0
• Fix Version/s: 1.17.0, 1.16.1, 1.15.4
• Component/s: Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Chesnay Schepler
• Reporter: Yun Gao
• Created: 09/Feb/22 07:44
• Updated: 25/May/23 12:34
• Resolved: 24/Nov/22 08:18

Description

Feb 09 02:22:52 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project flink-runtime: There are test failures.
Feb 09 02:22:52 [ERROR] 
Feb 09 02:22:52 [ERROR] Please refer to /__w/2/s/flink-runtime/target/surefire-reports for the individual test results.
Feb 09 02:22:52 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Feb 09 02:22:52 [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2 surefire8313405181432833392tmp surefire_2318122906826894495431tmp
Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
Feb 09 02:22:52 [ERROR] Process Exit Code: 239
Feb 09 02:22:52 [ERROR] Crashed tests:
Feb 09 02:22:52 [ERROR] org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
Feb 09 02:22:52 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2 surefire8313405181432833392tmp surefire_2318122906826894495431tmp
Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
Feb 09 02:22:52 [ERROR] Process Exit Code: 239
Feb 09 02:22:52 [ERROR] Crashed tests:
Feb 09 02:22:52 [ERROR] org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Feb 09 02:22:52 [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
Feb 09 02:22:52 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
Feb 09 02:22:52 [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
Feb 09 02:22:52 [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
Feb 09 02:22:52 [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=9164

Issue Links

• duplicates: FLINK-26106 BoundedSourceITCase failed due to JVM exits with code 239
• is duplicated by: FLINK-26632 JobManagerHAProcessFailureRecoveryITCase failed due to JVM exits with code 239
• is duplicated by: FLINK-25587 HiveCatalogITCase crashed on Azure with exit code 239
• is duplicated by: FLINK-30105 SubtaskExecutionAttemptAccumulatorsInfoTest failed with JVM exit code 239
• is related to: FLINK-29488 MetricRegistryImpl should implement AutoCloseableAsync
• relates to: FLINK-30153 YARNSessionCapacitySchedulerITCase fails with NoClassDefFoundError org/jboss/netty/channel/ExceptionEvent
• relates to: FLINK-31669 DynamicFilteringITCase failed fataly because of NoClassDefFoundError of akka/actor/SuppressedDeadLetter
• relates to: FLINK-30299 TaskManagerRunnerTest fails with 239 exit code (i.e. FatalExitExceptionHandler was called) NoClassDefFoundError: akka/remote/transport/netty/NettyFutureBridge$$anon$1
• links to: GitHub Pull Request #21346
• links to: GitHub Pull Request #21394