Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9812

Integration tests hang and timeout the entire PR build on jenkins

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: streams
    • Labels:

      Description

      Test org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] and possibly others have been spotted to hang indefinitely without entering a FAILED state, which resulted in the whole jenkins to time out hours later without a list of failures. 

      A suggested fix could possibly involve using a class wide Timeout rule with reasonable timeouts for integration tests, as described here: https://github.com/junit-team/junit4/wiki/Timeout-for-tests

      Snippet from the build log https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/console :

      org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] STARTED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbleToRunWithTwoSubtopologies[exactly_once_beta] PASSED 12:46:36 12:46:36 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] STARTED 15:12:14 Build timed out (after 270 minutes). Marking the build as aborted. 15:12:15 Build was aborted 15:12:15 [FINDBUGS] Skipping publisher since build result is ABORTED 15:12:15 Recording test results 15:12:15 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:15 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:15 15:12:15 org.apache.kafka.streams.integration.EosIntegrationTest > shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_beta] SKIPPED 15:12:16 15:12:16 > Task :streams:integrationTest FAILED 15:12:17 The message received from the daemon indicates that the daemon has disappeared. 15:12:17 Build request sent: Build{id=4f5e3086-ab55-48c5-a3a4-6f213565ac61, currentDir=/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.12} 15:12:17 Attempting to read last messages from the daemon log... 15:12:17 Daemon pid: 27019 15:12:17 log file: /home/jenkins/.gradle/daemon/5.6.2/daemon-27019.out.log 15:12:17 ----- Last 20 lines from daemon log file - daemon-27019.out.log ----- 15:12:17 at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:237) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:214) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:364) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87) 15:12:17 at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) 15:12:17 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) 15:12:17 at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) 15:12:17 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 15:12:17 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 15:12:17 at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) 15:12:17 at java.lang.Thread.run(Thread.java:748) 15:12:17 Caused by: java.lang.IllegalStateException: Shutdown in progress 15:12:17 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) 15:12:17 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) 15:12:17 at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:33) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:204) 15:12:17 at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:360) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) 15:12:17 at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) 15:12:17 ... 7 more 15:12:17 ----- End of the daemon log ----- 15:12:17 15:12:17 15:12:17 FAILURE: Build failed with an exception. 15:12:17 15:12:17 * What went wrong: 15:12:17 Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) 15:12:17 15:12:17 * Try: 15:12:17 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 15:12:17 15:12:17 * Get more help at https://help.gradle.org 15:12:17 > Task :tools:integrationTest 15:12:17 Test steps failed 15:12:17 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:17 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Adding one-line test results to commit status... 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Setting status of 45ec78a919d6220521b55ca63582d365664b85c1 to FAILURE with url https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1558/ and message: 'FAILURE 15:12:19 13289 tests run, 67 skipped, 0 failed.' 15:12:19 Using context: JDK 8 and Scala 2.12 15:12:19 Setting MAVEN_LATEST__HOME=/home/jenkins/tools/maven/latest/ 15:12:19 Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3 15:12:19 Finished: ABORTED
      

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vvcephei John Roesler
                Reporter:
                kkonstantine Konstantine Karantasis
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: