Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-9219

CI: multiple CI jobs fail with "timeout exceeded"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.15.0
    • Component/s: ci
    • Labels:
      None

      Description

      DistributedTestOpenJDK8, WindowsIntegrationTestOpenJDK8, and WindowsGfshDistributedTestOpenJDK11 jobs occasionally fail because they exceed their timeouts.
      These failures (or some of them) might be caused by GEODE-9079.

      In DistributedTestOpenJDK8 failures, artifacts show a thread stuck in ClassGraph code:

      "Test worker" #25 prio=5 os_prio=0 tid=0x00007fe464a96800 nid=0x5e waiting on condition [0x00007fe316ccc000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000d3ec4580> (a java.util.concurrent.FutureTask)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
              at java.util.concurrent.FutureTask.get(FutureTask.java:191)
              at io.github.classgraph.ClassGraph.scan(ClassGraph.java:1548)
              at io.github.classgraph.ClassGraph.scan(ClassGraph.java:1587)
              at org.apache.geode.management.internal.util.ClasspathScanLoadHelper.<init>(ClasspathScanLoadHelper.java:41)
              at org.apache.geode.management.internal.cli.CommandManager.loadCommands(CommandManager.java:185)
              at org.apache.geode.management.internal.cli.CommandManager.<init>(CommandManager.java:89)
              at org.apache.geode.management.internal.cli.remote.OnlineCommandProcessor.init(OnlineCommandProcessor.java:147)
              at org.apache.geode.internal.cache.GemFireCacheImpl.initializeServices(GemFireCacheImpl.java:1508)
              at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1435)
              at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
              - locked <0x00000000d0258b78> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl)
              - locked <0x00000000d0965568> (a java.lang.Class for org.apache.geode.internal.cache.InternalCacheBuilder)
              at org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
              at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
              at org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.createCache(JUnit4CacheTestCase.java:120)
              - locked <0x00000000d01d2b20> (a java.lang.Class for org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase)
              at org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:222)
              - locked <0x00000000d01d2b20> (a java.lang.Class for org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase)
              at org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:209)
              at org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.getCache(JUnit4CacheTestCase.java:196)
              at org.apache.geode.cache30.CacheXml66DUnitTest.testPartitionedRegionXML(CacheXml66DUnitTest.java:2737)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
              at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
              at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
              at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
              at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
              at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
              at org.apache.geode.test.dunit.rules.AbstractDistributedRule$1.evaluate(AbstractDistributedRule.java:59)
              at org.apache.geode.test.junit.rules.serializable.SerializableTemporaryFolder$1.evaluate(SerializableTemporaryFolder.java:130)
              at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
              at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
              at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
              at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
              at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
              at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
              at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
              at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
              at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
              at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
              at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
              at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
              at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
              at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
              at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
              at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
              at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
              at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
              at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
              at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
              at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
              at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
              at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
              at java.lang.Thread.run(Thread.java:748)
      

       

      Windows jobs seem to fail during rsync, so they might have a different root cause:

      BUILD SUCCESSFUL in 49s
       1 actionable task: 1 up-to-date
       + + waitrsync 
       -e 'ssh -i instance-data/sshkey -o ConnectTimeout=5 -o ConnectionAttempts=60 -o StrictHostKeyChecking=no' -ah geode@10.0.0.66:geode /tmp/build/1a3d1be6/geode-results/
       + rsync -e 'ssh -i instance-data/sshkey -o ConnectTimeout=5 -o ConnectionAttempts=60 -o StrictHostKeyChecking=no' -ah geode@10.0.0.66:.gradle /tmp/build/1a3d1be6/geode-results/geode/.gradle_logs
      real 4m25.424s
       user 0m28.586s
       sys 0m25.156s
       rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642) [generator=3.1.3]
       rsync: [generator] write error: Broken pipe (32)
       rsync error: received SIGUSR1 (code 19) at main.c(1440) [receiver=3.1.3]
      timeout exceeded

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ukohlmeyer Udo Kohlmeyer
                Reporter:
                kaslami Kamilla Aslami
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: