Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21929

flink-statebackend-rocksdb crashes with Error occurred in starting fork

    XMLWordPrintableJSON

Details

    Description

      https://dev.azure.com/rmetzger/Flink/_build/results?buildId=9001&view=results

      2021-03-23T09:11:12.1861967Z [INFO] BUILD FAILURE
2021-03-23T09:11:12.1863007Z [INFO] ------------------------------------------------------------------------
2021-03-23T09:11:12.1863492Z [INFO] Total time: 42:35 min
2021-03-23T09:11:12.1864171Z [INFO] Finished at: 2021-03-23T09:11:12+00:00
2021-03-23T09:11:12.8003245Z [INFO] Final Memory: 137M/806M
2021-03-23T09:11:12.8006310Z [INFO] ------------------------------------------------------------------------
2021-03-23T09:11:12.8082409Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project flink-statebackend-rocksdb_2.11: There are test failures.
2021-03-23T09:11:12.8086652Z [ERROR] 
2021-03-23T09:11:12.8092462Z [ERROR] Please refer to /__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports for the individual test results.
2021-03-23T09:11:12.8096948Z [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
2021-03-23T09:11:12.8101388Z [ERROR] ExecutionException Error occurred in starting fork, check output in log
2021-03-23T09:11:12.8105868Z [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException Error occurred in starting fork, check output in log
2021-03-23T09:11:12.8110518Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
2021-03-23T09:11:12.8115518Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:382)
2021-03-23T09:11:12.8120811Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
2021-03-23T09:11:12.8126356Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
2021-03-23T09:11:12.8127129Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
2021-03-23T09:11:12.8131291Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
2021-03-23T09:11:12.8132369Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
2021-03-23T09:11:12.8133397Z [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
2021-03-23T09:11:12.8134116Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
2021-03-23T09:11:12.8134793Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
2021-03-23T09:11:12.8135621Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
2021-03-23T09:11:12.8136323Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
2021-03-23T09:11:12.8141570Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
2021-03-23T09:11:12.8142374Z [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
2021-03-23T09:11:12.8145665Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
2021-03-23T09:11:12.8146407Z [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
2021-03-23T09:11:12.8148835Z [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
2021-03-23T09:11:12.8151299Z [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
2021-03-23T09:11:12.8152244Z [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
2021-03-23T09:11:12.8152806Z [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
2021-03-23T09:11:12.8155818Z [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-03-23T09:11:12.8159757Z [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-03-23T09:11:12.8177288Z [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-03-23T09:11:12.8178021Z [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
2021-03-23T09:11:12.8179802Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
2021-03-23T09:11:12.8183929Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
2021-03-23T09:11:12.8187563Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
2021-03-23T09:11:12.8192413Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
2021-03-23T09:11:12.8196538Z [ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log
2021-03-23T09:11:12.8201660Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:622)
2021-03-23T09:11:12.8203999Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
2021-03-23T09:11:12.8204879Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:371)
2021-03-23T09:11:12.8205665Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:347)
2021-03-23T09:11:12.8206513Z [ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-03-23T09:11:12.8207169Z [ERROR] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-03-23T09:11:12.8209376Z [ERROR] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-03-23T09:11:12.8209955Z [ERROR] at java.lang.Thread.run(Thread.java:748)
2021-03-23T09:11:12.8211107Z [ERROR] -> [Help 1]
2021-03-23T09:11:12.8211559Z [ERROR] 
2021-03-23T09:11:12.8309016Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2021-03-23T09:11:12.8310211Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2021-03-23T09:11:12.8311401Z [ERROR] 
2021-03-23T09:11:12.8311976Z [ERROR] For more information about the errors and possible solutions, please read the following articles:
2021-03-23T09:11:12.8312635Z [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
2021-03-23T09:11:12.8313300Z [ERROR] 
2021-03-23T09:11:12.8314248Z [ERROR] After correcting the problems, you can resume the build with the command
2021-03-23T09:11:12.8315462Z [ERROR]   mvn <goals> -rf :flink-statebackend-rocksdb_2.11
2021-03-23T09:11:12.8687280Z Process exited with EXIT CODE: 1.
2021-03-23T09:11:12.8687830Z Trying to KILL watchdog (359).
2021-03-23T09:11:12.8696302Z /__w/1/s/tools/ci/watchdog.sh: line 100:   359 Terminated              watchdog
2021-03-23T09:11:16.4147696Z Searching for .dump, .dumpstream and related files in '/__w/1/s'
2021-03-23T09:11:21.6569652Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6645588Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6683479Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6726888Z Moving '/__w/1/s/flink-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6772031Z Moving '/__w/1/s/flink-metrics/flink-metrics-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6807831Z Moving '/__w/1/s/flink-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6847619Z Moving '/__w/1/s/flink-runtime-web/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:21.6882147Z Moving '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/core.18126' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1251457Z Moving '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1287286Z Moving '/__w/1/s/flink-test-utils-parent/flink-test-utils/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1322850Z Moving '/__w/1/s/flink-clients/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1360757Z Moving '/__w/1/s/flink-streaming-java/java_pid5009.hprof' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1421701Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1460325Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1492941Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1523043Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1554119Z Moving '/__w/1/s/flink-optimizer/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1589618Z Moving '/__w/1/s/flink-scala/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
2021-03-23T09:11:22.1638143Z Compressing debug files

      I'm not sure if this includes one or two issues.

      One issue seems to be in the flink-streaming-java tests:

      # Created at 2021-03-23T09:03:49.666
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'java.lang.OutOfMemoryError: Java heap space'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.OutOfMemoryError: Java heap space'.
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
	at java.lang.Thread.run(Thread.java:748)


# Created at 2021-03-23T09:03:49.667
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'Dumping heap to java_pid5009.hprof ...'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Dumping heap to java_pid5009.hprof ...'.
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
	at java.lang.Thread.run(Thread.java:748)


# Created at 2021-03-23T09:03:49.693
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'Heap dump file created [2834354 bytes in 0.027 secs]'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Heap dump file created [2834354 bytes in 0.027 secs]'.
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
	at java.lang.Thread.run(Thread.java:748)

      There's a coredump included:


      Looks like the testInitialSizeCompoutation test is causing the / a failure.

      The second issue is (which is reported as a test failure): 

      # Created at 2021-03-23T09:09:39.376
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError

# Created at 2021-03-23T09:09:39.535
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError

# Created at 2021-03-23T09:11:10.037
pure virtual method called

# Created at 2021-03-23T09:11:10.037
terminate called without an active exception

# Created at 2021-03-23T09:11:12.170
Aborted (core dumped)

      From the coredump, I see the following:

      (gdb) where
#0  0x00007f9343508438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f934350a03a in __GI_abort () at abort.c:89
#2  0x00007f9341b8e84d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f9341b8c6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f9341b8c701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f9341b8d23f in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f92ed39c0d5 in rocksdb::DBImpl::CloseHelper() () from /tmp/librocksdbjni8334465068904677424.so
#7  0x00007f92ed3a717b in rocksdb::DBImpl::~DBImpl() () from /tmp/librocksdbjni8334465068904677424.so
#8  0x00007f92ed3a7451 in rocksdb::DBImpl::~DBImpl() () from /tmp/librocksdbjni8334465068904677424.so
#9  0x00007f932d801aa8 in ?? ()
#10 0x0000000085677fe8 in ?? ()
#11 0x00007f931c8da910 in ?? ()
#12 0x00007f931c8da968 in ?? ()
#13 0x00007f932d007ffd in ?? ()
#14 0x0000000000000000 in ?? ()

      Attachments

        1. image-2021-03-23-13-18-41-836.png
          72 kB
          Robert Metzger

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. FLINK-21965 Surefire plugin seems to have failed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. FLINK-22025 SurefireBooterForkException: ExecutionException Error occurred in starting fork

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #15380

          Activity

            People

              yunta Yun Tang
              rmetzger Robert Metzger
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: