Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21929

flink-statebackend-rocksdb crashes with Error occurred in starting fork

    XMLWordPrintableJSON

    Details

      Description

      https://dev.azure.com/rmetzger/Flink/_build/results?buildId=9001&view=results

      2021-03-23T09:11:12.1861967Z [INFO] BUILD FAILURE
      2021-03-23T09:11:12.1863007Z [INFO] ------------------------------------------------------------------------
      2021-03-23T09:11:12.1863492Z [INFO] Total time: 42:35 min
      2021-03-23T09:11:12.1864171Z [INFO] Finished at: 2021-03-23T09:11:12+00:00
      2021-03-23T09:11:12.8003245Z [INFO] Final Memory: 137M/806M
      2021-03-23T09:11:12.8006310Z [INFO] ------------------------------------------------------------------------
      2021-03-23T09:11:12.8082409Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project flink-statebackend-rocksdb_2.11: There are test failures.
      2021-03-23T09:11:12.8086652Z [ERROR] 
      2021-03-23T09:11:12.8092462Z [ERROR] Please refer to /__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports for the individual test results.
      2021-03-23T09:11:12.8096948Z [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
      2021-03-23T09:11:12.8101388Z [ERROR] ExecutionException Error occurred in starting fork, check output in log
      2021-03-23T09:11:12.8105868Z [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException Error occurred in starting fork, check output in log
      2021-03-23T09:11:12.8110518Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
      2021-03-23T09:11:12.8115518Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:382)
      2021-03-23T09:11:12.8120811Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
      2021-03-23T09:11:12.8126356Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
      2021-03-23T09:11:12.8127129Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
      2021-03-23T09:11:12.8131291Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
      2021-03-23T09:11:12.8132369Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
      2021-03-23T09:11:12.8133397Z [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
      2021-03-23T09:11:12.8134116Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
      2021-03-23T09:11:12.8134793Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      2021-03-23T09:11:12.8135621Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      2021-03-23T09:11:12.8136323Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
      2021-03-23T09:11:12.8141570Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
      2021-03-23T09:11:12.8142374Z [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
      2021-03-23T09:11:12.8145665Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
      2021-03-23T09:11:12.8146407Z [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
      2021-03-23T09:11:12.8148835Z [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
      2021-03-23T09:11:12.8151299Z [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
      2021-03-23T09:11:12.8152244Z [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
      2021-03-23T09:11:12.8152806Z [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
      2021-03-23T09:11:12.8155818Z [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2021-03-23T09:11:12.8159757Z [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      2021-03-23T09:11:12.8177288Z [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      2021-03-23T09:11:12.8178021Z [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
      2021-03-23T09:11:12.8179802Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
      2021-03-23T09:11:12.8183929Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
      2021-03-23T09:11:12.8187563Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
      2021-03-23T09:11:12.8192413Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
      2021-03-23T09:11:12.8196538Z [ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log
      2021-03-23T09:11:12.8201660Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:622)
      2021-03-23T09:11:12.8203999Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
      2021-03-23T09:11:12.8204879Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:371)
      2021-03-23T09:11:12.8205665Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:347)
      2021-03-23T09:11:12.8206513Z [ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-03-23T09:11:12.8207169Z [ERROR] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-03-23T09:11:12.8209376Z [ERROR] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-03-23T09:11:12.8209955Z [ERROR] at java.lang.Thread.run(Thread.java:748)
      2021-03-23T09:11:12.8211107Z [ERROR] -> [Help 1]
      2021-03-23T09:11:12.8211559Z [ERROR] 
      2021-03-23T09:11:12.8309016Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      2021-03-23T09:11:12.8310211Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      2021-03-23T09:11:12.8311401Z [ERROR] 
      2021-03-23T09:11:12.8311976Z [ERROR] For more information about the errors and possible solutions, please read the following articles:
      2021-03-23T09:11:12.8312635Z [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
      2021-03-23T09:11:12.8313300Z [ERROR] 
      2021-03-23T09:11:12.8314248Z [ERROR] After correcting the problems, you can resume the build with the command
      2021-03-23T09:11:12.8315462Z [ERROR]   mvn <goals> -rf :flink-statebackend-rocksdb_2.11
      2021-03-23T09:11:12.8687280Z Process exited with EXIT CODE: 1.
      2021-03-23T09:11:12.8687830Z Trying to KILL watchdog (359).
      2021-03-23T09:11:12.8696302Z /__w/1/s/tools/ci/watchdog.sh: line 100:   359 Terminated              watchdog
      2021-03-23T09:11:16.4147696Z Searching for .dump, .dumpstream and related files in '/__w/1/s'
      2021-03-23T09:11:21.6569652Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6645588Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6683479Z Moving '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6726888Z Moving '/__w/1/s/flink-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6772031Z Moving '/__w/1/s/flink-metrics/flink-metrics-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6807831Z Moving '/__w/1/s/flink-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6847619Z Moving '/__w/1/s/flink-runtime-web/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:21.6882147Z Moving '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/core.18126' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1251457Z Moving '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1287286Z Moving '/__w/1/s/flink-test-utils-parent/flink-test-utils/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1322850Z Moving '/__w/1/s/flink-clients/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1360757Z Moving '/__w/1/s/flink-streaming-java/java_pid5009.hprof' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1421701Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1460325Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1492941Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1523043Z Moving '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1554119Z Moving '/__w/1/s/flink-optimizer/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1589618Z Moving '/__w/1/s/flink-scala/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream' to target directory ('/__w/_temp/debug_files/')
      2021-03-23T09:11:22.1638143Z Compressing debug files
      

      I'm not sure if this includes one or two issues.

      One issue seems to be in the flink-streaming-java tests:

      # Created at 2021-03-23T09:03:49.666
      Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'java.lang.OutOfMemoryError: Java heap space'.
      java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.OutOfMemoryError: Java heap space'.
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
      	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
      	at java.lang.Thread.run(Thread.java:748)
      
      
      # Created at 2021-03-23T09:03:49.667
      Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'Dumping heap to java_pid5009.hprof ...'.
      java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Dumping heap to java_pid5009.hprof ...'.
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
      	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
      	at java.lang.Thread.run(Thread.java:748)
      
      
      # Created at 2021-03-23T09:03:49.693
      Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'Heap dump file created [2834354 bytes in 0.027 secs]'.
      java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Heap dump file created [2834354 bytes in 0.027 secs]'.
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
      	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
      	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
      	at java.lang.Thread.run(Thread.java:748)
      

      There's a coredump included:


      Looks like the testInitialSizeCompoutation test is causing the / a failure.

      The second issue is (which is reported as a test failure):

      # Created at 2021-03-23T09:09:39.376
      Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
      
      # Created at 2021-03-23T09:09:39.535
      Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
      
      # Created at 2021-03-23T09:11:10.037
      pure virtual method called
      
      # Created at 2021-03-23T09:11:10.037
      terminate called without an active exception
      
      # Created at 2021-03-23T09:11:12.170
      Aborted (core dumped)
      

      From the coredump, I see the following:

      (gdb) where
      #0  0x00007f9343508438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
      #1  0x00007f934350a03a in __GI_abort () at abort.c:89
      #2  0x00007f9341b8e84d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #3  0x00007f9341b8c6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #4  0x00007f9341b8c701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #5  0x00007f9341b8d23f in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #6  0x00007f92ed39c0d5 in rocksdb::DBImpl::CloseHelper() () from /tmp/librocksdbjni8334465068904677424.so
      #7  0x00007f92ed3a717b in rocksdb::DBImpl::~DBImpl() () from /tmp/librocksdbjni8334465068904677424.so
      #8  0x00007f92ed3a7451 in rocksdb::DBImpl::~DBImpl() () from /tmp/librocksdbjni8334465068904677424.so
      #9  0x00007f932d801aa8 in ?? ()
      #10 0x0000000085677fe8 in ?? ()
      #11 0x00007f931c8da910 in ?? ()
      #12 0x00007f931c8da968 in ?? ()
      #13 0x00007f932d007ffd in ?? ()
      #14 0x0000000000000000 in ?? ()
      

        Attachments

        1. image-2021-03-23-13-18-41-836.png
          72 kB
          Robert Metzger

          Issue Links

            Activity

              People

              • Assignee:
                yunta Yun Tang
                Reporter:
                rmetzger Robert Metzger
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: