Infrastructure
  1. Infrastructure
  2. INFRA-4272

Camel.trunk.fulltest.windows - Still Failing

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: Jenkins
    • Labels:
      None
    • Environment:
      Windows

      Description

      Since build 166 (Dec 26, 2011 6:01:05 PM) we hit again the issue that camel-core-XXX.jar cannot be deleted. We see this issue so often. Do you have any idea what's going wrong here? May other projects had the same issue and you could help them (permanently)? I know other projects like CXF which disabled the windows build, but this doesn't look like a solution for me...
      Any help or suggestion is welcome.

      See https://builds.apache.org/job/Camel.trunk.fulltest.windows/ for details.

      Stacktrace:
      mavenExecutionResult exceptions not empty
      message : Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.4.1:clean (default-clean) on project camel-core: Failed to clean project: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
      cause : Failed to clean project: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
      Stack trace :
      org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.4.1:clean (default-clean) on project camel-core: Failed to clean project: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
      at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
      at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
      at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:158)
      at hudson.maven.Maven3Builder.call(Maven3Builder.java:104)
      at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:287)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to clean project: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
      at org.apache.maven.plugin.clean.CleanMojo.execute(CleanMojo.java:188)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
      ... 27 more
      Caused by: java.io.IOException: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
      at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:227)
      at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:186)
      at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:154)
      at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:115)
      at org.apache.maven.plugin.clean.CleanMojo.execute(CleanMojo.java:168)
      ... 29 more
      1. trace.txt
        8 kB
        Brett Porter

        Issue Links

          Activity

          Hide
          Brett Porter added a comment -
          resolving, as we should be addressing via the builds@apache.org list if a build-specific issue comes up
          Show
          Brett Porter added a comment - resolving, as we should be addressing via the builds@apache.org list if a build-specific issue comes up
          Hide
          Babak Vahdat added a comment -
          @Brett

          Thanks for the hint which made it now clear to me. The last three builds failed again (247-249) as the build 246 now has a grey ball on it (was killed manually or because of reboot etc.):

          {code}
          Running org.apache.camel.component.cxf.CxfGreeterWSDLOnlyRouterTest
          Build timed out (after *365* minutes). Marking the build as failed.
          Process 1328856829304 is killed.
          [ERROR] There are test failures.
          {code}

          Could you please (as you already recommended) intervene so that we try another build on Windows. AFAIK the issue we're suffering from in Camel tests are (dynamically calculated) ports we try to make use of while running the unit-tests and most probably there're some improvement on this corner. As you can imagine reproducing the same issue on one's own Windows box is pretty difficult.

          Thanks for your help!
          Show
          Babak Vahdat added a comment - @Brett Thanks for the hint which made it now clear to me. The last three builds failed again (247-249) as the build 246 now has a grey ball on it (was killed manually or because of reboot etc.): {code} Running org.apache.camel.component.cxf.CxfGreeterWSDLOnlyRouterTest Build timed out (after *365* minutes). Marking the build as failed. Process 1328856829304 is killed. [ERROR] There are test failures. {code} Could you please (as you already recommended) intervene so that we try another build on Windows. AFAIK the issue we're suffering from in Camel tests are (dynamically calculated) ports we try to make use of while running the unit-tests and most probably there're some improvement on this corner. As you can imagine reproducing the same issue on one's own Windows box is pretty difficult. Thanks for your help!
          Hide
          Brett Porter added a comment -
          What probably happened was that someone killed the stuck process, or rebooted the machine. The concern is not the "failed to delete" error, we know why that happens.

          The concerns are:
          1) the intermittent hangs, such as in build #216 building the CXF module
          2) that the build takes so long and is more prone to getting killed

          Whenever you see a grey ball on a build, it is likely that a subsequent build will fail with the "failed to delete" error and require administrative intervention.
          Show
          Brett Porter added a comment - What probably happened was that someone killed the stuck process, or rebooted the machine. The concern is not the "failed to delete" error, we know why that happens. The concerns are: 1) the intermittent hangs, such as in build #216 building the CXF module 2) that the build takes so long and is more prone to getting killed Whenever you see a grey ball on a build, it is likely that a subsequent build will fail with the "failed to delete" error and require administrative intervention.
          Hide
          Babak Vahdat added a comment -
          interestingly the again-and-again-repeating Maven issue on Windows:

          Caused by: java.io.IOException: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar

          did *not* happen anymore by the last build of today morning, so that the behaviour is to my understanding non-deterministic:

          https://builds.apache.org/job/Camel.trunk.fulltest.windows/231/console

          I've got no clue which "zombie" process or what exactly inhibits Maven to delete that jar (from time to time) which happened again by the previous two builds:

          https://builds.apache.org/job/Camel.trunk.fulltest.windows/230/console
          https://builds.apache.org/job/Camel.trunk.fulltest.windows/229/console
          Show
          Babak Vahdat added a comment - interestingly the again-and-again-repeating Maven issue on Windows: Caused by: java.io.IOException: Failed to delete F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar did *not* happen anymore by the last build of today morning, so that the behaviour is to my understanding non-deterministic: https://builds.apache.org/job/Camel.trunk.fulltest.windows/231/console I've got no clue which "zombie" process or what exactly inhibits Maven to delete that jar (from time to time) which happened again by the previous two builds: https://builds.apache.org/job/Camel.trunk.fulltest.windows/230/console https://builds.apache.org/job/Camel.trunk.fulltest.windows/229/console
          Hide
          Christian Müller added a comment -
          Hmmm, it works on my Windows 7 box...
          I will ask Babak to have a look on it. I will be offline for the next 16 days...
          Show
          Christian Müller added a comment - Hmmm, it works on my Windows 7 box... I will ask Babak to have a look on it. I will be offline for the next 16 days...
          Hide
          Brett Porter added a comment -
          It looks like it got stuck in CXFBeanTest, so you'll need to look into why that hung:
          https://builds.apache.org/job/Camel.trunk.fulltest.windows/216/org.apache.camel$camel-cxf/console

          Any time a build hangs to the point it needs to be killed, the subsequent builds are not going to succeed - unfortunately there's not much I can do about that.

          I'll clear the stuck process again now.
          Show
          Brett Porter added a comment - It looks like it got stuck in CXFBeanTest, so you'll need to look into why that hung: https://builds.apache.org/job/Camel.trunk.fulltest.windows/216/org.apache.camel$camel-cxf/console Any time a build hangs to the point it needs to be killed, the subsequent builds are not going to succeed - unfortunately there's not much I can do about that. I'll clear the stuck process again now.
          Hide
          Christian Müller added a comment -
          Could someone please have again a look on it?
          Show
          Christian Müller added a comment - Could someone please have again a look on it?
          Hide
          Babak Vahdat added a comment - - edited
          Again we're still suffering from the same issue on Windows:
          https://builds.apache.org/job/Camel.trunk.fulltest.windows/224/

          Where the top most SVN revision on it is 1236680, which is *exactly* the same as:
          https://builds.apache.org/job/Camel.trunk.fulltest/672/

          on a non-Windows which passed well with *full test* as well. So to my understanding there're some Windows "zombie" processes hanging around inhibiting Maven to delete:
          F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar
          again and again and again!
          Show
          Babak Vahdat added a comment - - edited Again we're still suffering from the same issue on Windows: https://builds.apache.org/job/Camel.trunk.fulltest.windows/224/ Where the top most SVN revision on it is 1236680, which is *exactly* the same as: https://builds.apache.org/job/Camel.trunk.fulltest/672/ on a non-Windows which passed well with *full test* as well. So to my understanding there're some Windows "zombie" processes hanging around inhibiting Maven to delete: F:\hudson\hudson-slave\workspace\Camel.trunk.fulltest.windows\camel-trunk\camel-core\target\camel-core-2.10-SNAPSHOT.jar again and again and again!
          Hide
          Brett Porter added a comment -
          That was a temporary issue. It seems the build is now succeeding. Thanks!
          Show
          Brett Porter added a comment - That was a temporary issue. It seems the build is now succeeding. Thanks!
          Hide
          Brett Porter added a comment -
          This time it got stuck much earlier, in Camel :: CXF. Trying again to see if that's a one-off problem...
          Show
          Brett Porter added a comment - This time it got stuck much earlier, in Camel :: CXF. Trying again to see if that's a one-off problem...
          Hide
          Christian Müller added a comment -
          Hello Brett!
          We could solve this problem [CAMEL-4924|https://issues.apache.org/jira/browse/CAMEL-4924]. I started a build manually to see how it works (the last two builds aborted in camel-jms). No idea why...

          Thanks for your help,
          Christian
          Show
          Christian Müller added a comment - Hello Brett! We could solve this problem [ CAMEL-4924 | https://issues.apache.org/jira/browse/CAMEL-4924 ]. I started a build manually to see how it works (the last two builds aborted in camel-jms). No idea why... Thanks for your help, Christian
          Hide
          Brett Porter added a comment -
          attaching stack trace
          Show
          Brett Porter added a comment - attaching stack trace
          Hide
          Brett Porter added a comment -
          I'll need one of the other admins to help get a thread dump from the stuck process, then I'll kill it again.
          Show
          Brett Porter added a comment - I'll need one of the other admins to help get a thread dump from the stuck process, then I'll kill it again.
          Hide
          Brett Porter added a comment -
          I'll give it more time, but it looks to have the same problem. It might be worth noting that the test before it is failing - maybe that leaves bad state behind?
          Show
          Brett Porter added a comment - I'll give it more time, but it looks to have the same problem. It might be worth noting that the test before it is failing - maybe that leaves bad state behind?
          Hide
          Brett Porter added a comment -
          building now
          Show
          Brett Porter added a comment - building now
          Hide
          Christian Müller added a comment -
          I applied a patch which should fix this issue. But at the moment the build failed because camel-core-2.10-SNAPSHOT.jar cannot be deleted. Could you please have again a look on it?

          Thanks for your help,
          Christian
          Show
          Christian Müller added a comment - I applied a patch which should fix this issue. But at the moment the build failed because camel-core-2.10-SNAPSHOT.jar cannot be deleted. Could you please have again a look on it? Thanks for your help, Christian
          Hide
          Brett Porter added a comment -
          No, it's because this build froze and held onto it in the unit tests:
          https://builds.apache.org/view/A-F/view/Camel/job/Camel.trunk.fulltest.windows/203/console

          You'll ned to resolve why that test freezes, then it should be fine. Can you reproduce that somewhere? Let me know if you need any help looking at something on the build server.
          Show
          Brett Porter added a comment - No, it's because this build froze and held onto it in the unit tests: https://builds.apache.org/view/A-F/view/Camel/job/Camel.trunk.fulltest.windows/203/console You'll ned to resolve why that test freezes, then it should be fine. Can you reproduce that somewhere? Let me know if you need any help looking at something on the build server.
          Hide
          Christian Müller added a comment -
          Thank you Brett!
          I'm wondering why the camel-core-XXX.jar can (still) not be removed from the maven-clean-plugin... May there is another demon thread?
          Show
          Christian Müller added a comment - Thank you Brett! I'm wondering why the camel-core-XXX.jar can (still) not be removed from the maven-clean-plugin... May there is another demon thread?
          Hide
          Brett Porter added a comment -
          It seems to have blocked in this test:
          Running org.apache.camel.component.mina2.Mina2TcpWithInOutUsingPlainSocketTest
          Show
          Brett Porter added a comment - It seems to have blocked in this test: Running org.apache.camel.component.mina2.Mina2TcpWithInOutUsingPlainSocketTest
          Hide
          Brett Porter added a comment -
          - killed a stuck Camel build
          - deleted the file as an administrator
          - running build again now
          Show
          Brett Porter added a comment - - killed a stuck Camel build - deleted the file as an administrator - running build again now
          Hide
          Christian Müller added a comment -
          Someone not to busy to help us?
          Show
          Christian Müller added a comment - Someone not to busy to help us?
          Hide
          Christian Müller added a comment -
          Could you please kill the demon thread which locks the file? Until the next build...
          Show
          Christian Müller added a comment - Could you please kill the demon thread which locks the file? Until the next build...

            People

            • Assignee:
              Brett Porter
              Reporter:
              Christian Müller
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development