Whirr
  1. Whirr
  2. WHIRR-228 Run integration tests using a CI server
  3. WHIRR-422

Integration tests should fail or succeed in a limited amount of time

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:

      Description

      We should make sure that the integration tests either fail or succeed in a limited (and short) amount of time. Some of them are blocking if the remote service is not available.

      1. WHIRR-422.patch
        32 kB
        Andrei Savu

        Activity

        Hide
        Andrei Savu added a comment -

        Committed to trunk.

        Show
        Andrei Savu added a comment - Committed to trunk.
        Hide
        David Alves added a comment -

        +1 the patch applies cleanly and I tested the annotation ad-hoc.

        Show
        David Alves added a comment - +1 the patch applies cleanly and I tested the annotation ad-hoc.
        Hide
        Andrei Savu added a comment -

        ... it works and the @After method is executed.

        Show
        Andrei Savu added a comment - ... it works and the @After method is executed.
        Hide
        Andrei Savu added a comment -

        I think we should commit. I have tested with Zookeeper on cloudservers-uk and the annotation works. The failure message is something like "test timed out after 10 milliseconds".

        Show
        Andrei Savu added a comment - I think we should commit. I have tested with Zookeeper on cloudservers-uk and the annotation works. The failure message is something like "test timed out after 10 milliseconds".
        Hide
        Andrei Savu added a comment -

        Based on this question I think the annotation method should work:
        http://stackoverflow.com/questions/5102083/junit4-cleanup-on-test-timeout

        Show
        Andrei Savu added a comment - Based on this question I think the annotation method should work: http://stackoverflow.com/questions/5102083/junit4-cleanup-on-test-timeout
        Hide
        Andrei Savu added a comment -

        If we can't do this with junit maybe we should investigate testng.

        Show
        Andrei Savu added a comment - If we can't do this with junit maybe we should investigate testng.
        Hide
        David Alves added a comment -

        I just looked into the surefire code and indeed it is not what we want. All it does is execute a command line with timeout (forked vm), no tear-down is ever called.

        however this got me worried that the timeout won't work either: http://stackoverflow.com/questions/2689217/maven2-junit-timeout-annotation-doesnt-work

        I'll try your patch anyhow.

        Show
        David Alves added a comment - I just looked into the surefire code and indeed it is not what we want. All it does is execute a command line with timeout (forked vm), no tear-down is ever called. however this got me worried that the timeout won't work either: http://stackoverflow.com/questions/2689217/maven2-junit-timeout-annotation-doesnt-work I'll try your patch anyhow.
        Hide
        Andrei Savu added a comment -

        forkedProcessTimeoutInSeconds is not what we want. I will check to see if the annotation works as expected.

        Show
        Andrei Savu added a comment - forkedProcessTimeoutInSeconds is not what we want. I will check to see if the annotation works as expected.
        Hide
        David Alves added a comment - - edited

        Andrei: that was the idea I mentioned initially, the thing is I'm not sure tearDown() will actually be called (I'm inclined to say it won't) so this would make the tests fail but the clusters would be dangling, that's why I welcomed your idea.

        Anyhow its worth a try. I'll do that and report later.

        (btw you found an old maven 1.x page but there is an equivalent for the itest plugin whirr's using http://maven.apache.org/plugins/maven-failsafe-plugin/integration-test-mojo.html)

        Show
        David Alves added a comment - - edited Andrei: that was the idea I mentioned initially, the thing is I'm not sure tearDown() will actually be called (I'm inclined to say it won't) so this would make the tests fail but the clusters would be dangling, that's why I welcomed your idea. Anyhow its worth a try. I'll do that and report later. (btw you found an old maven 1.x page but there is an equivalent for the itest plugin whirr's using http://maven.apache.org/plugins/maven-failsafe-plugin/integration-test-mojo.html )
        Hide
        Andrei Savu added a comment -

        David do you want to give it a try? You've got a better understanding of how maven works - we should add the options when running mvn verify -Pintegration ...

        Show
        Andrei Savu added a comment - David do you want to give it a try? You've got a better understanding of how maven works - we should add the options when running mvn verify -Pintegration ...
        Hide
        Andrei Savu added a comment -

        Actually after checking http://maven.apache.org/maven-1.x/plugins/test/properties.html I think the right way of doing this is by configuration. maven.junit.timeout & maven.junit.fork

        Show
        Andrei Savu added a comment - Actually after checking http://maven.apache.org/maven-1.x/plugins/test/properties.html I think the right way of doing this is by configuration. maven.junit.timeout & maven.junit.fork
        Hide
        Andrei Savu added a comment -

        In this patch I've added a 15 mins timeout to all integration tests. David what do you think?

        Show
        Andrei Savu added a comment - In this patch I've added a 15 mins timeout to all integration tests. David what do you think?
        Hide
        Andrei Savu added a comment -

        If we get this in I will enable integration testing on a private Jenkins instance.

        Show
        Andrei Savu added a comment - If we get this in I will enable integration testing on a private Jenkins instance.
        Hide
        Andrei Savu added a comment -

        still would will we define the timeouts? educated guesses per service? might change from service to service, provider to provider, hardware to hardware...?

        Let's start with something like max ~15 minutes per test. This is more than enough for amazon & rackspace.

        Show
        Andrei Savu added a comment - still would will we define the timeouts? educated guesses per service? might change from service to service, provider to provider, hardware to hardware...? Let's start with something like max ~15 minutes per test. This is more than enough for amazon & rackspace.
        Hide
        David Alves added a comment -

        of course, makes sense, forgot about that

        still would will we define the timeouts? educated guesses per service? might change from service to service, provider to provider, hardware to hardware...?

        Show
        David Alves added a comment - of course, makes sense, forgot about that still would will we define the timeouts? educated guesses per service? might change from service to service, provider to provider, hardware to hardware...?
        Hide
        Andrei Savu added a comment -

        I think the best way to do this is by adding a timeout [1] specification to tests. This should still allow us to run tearDown code as expected. What do you think?

        [1] http://junit.sourceforge.net/javadoc/org/junit/Test.html

        Show
        Andrei Savu added a comment - I think the best way to do this is by adding a timeout [1] specification to tests. This should still allow us to run tearDown code as expected. What do you think? [1] http://junit.sourceforge.net/javadoc/org/junit/Test.html
        Hide
        David Alves added a comment -

        ok, so what I was saying doesn't make sense. it has to be something that is integrated into the lifecycle of the test as it is the test that knows the name of the cluster that was launched right?

        I'm not seeing any trivial way to do this other than creating a parent class for all tests or wrapping the itests into another class (suite?).

        Show
        David Alves added a comment - ok, so what I was saying doesn't make sense. it has to be something that is integrated into the lifecycle of the test as it is the test that knows the name of the cluster that was launched right? I'm not seeing any trivial way to do this other than creating a parent class for all tests or wrapping the itests into another class (suite?).
        Hide
        Andrei Savu added a comment -

        Maybe. The problem is making sure that no expensive cloud resources remain allocated.

        Show
        Andrei Savu added a comment - Maybe. The problem is making sure that no expensive cloud resources remain allocated.
        Hide
        David Alves added a comment -

        I can set this up with maven so that maven failsafe plugin (the test running plugin for itests) always forks for each test and fails only on itests and on a (common) timeout.
        Would this be acceptable?

        Show
        David Alves added a comment - I can set this up with maven so that maven failsafe plugin (the test running plugin for itests) always forks for each test and fails only on itests and on a (common) timeout. Would this be acceptable?

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Andrei Savu
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development