Derby
  1. Derby
  2. DERBY-5103

ProcedureInTriggerTest depends on implicit ordering of test cases

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.8.1.2
    • Fix Version/s: 10.8.1.2
    • Component/s: Test
    • Labels:
      None
    • Bug behavior facts:
      Regression Test Failure

      Description

      If for example the test case testTriggerContainsSql is executed as the first test case, it will fail with the following error:

      1) testTriggerContainsSql(org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTest)junit.framework.AssertionFailedError: expected:<2> but was:<0>
      at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTest.checkAndResetGetConnectionProcCount(ProcedureInTriggerTest.java:514)
      at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTest.testTriggerContainsSql(ProcedureInTriggerTest.java:120)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:112)
      at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
      at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
      at junit.extensions.TestSetup.run(TestSetup.java:25)
      at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)

      The test should either make the order of the test cases explicit or make the test cases independent of the order in which they run.

      Seen in the nightly tests, for example here: http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/testing/testlog/lin/1076682-suitesAll_diff.txt

      1. derby-5103-1a.diff
        3 kB
        Knut Anders Hatlen
      2. derby-5103-2a.diff
        0.9 kB
        Knut Anders Hatlen
      3. rjall.out
        696 kB
        Lily Wei

        Activity

        Hide
        Rick Hillegas added a comment -

        I ran this test on linux on Java 7 build 132. I could not reproduce the unusual ordering of test cases which is turning up in the Java 7 test runs on Solaris. I ran the test 10 times without errors. I also tried running some other tests which show instability on Solaris on Java 7. Again, I am not able to reproduce these instabilities on linux on Java 7 build 132.

        Show
        Rick Hillegas added a comment - I ran this test on linux on Java 7 build 132. I could not reproduce the unusual ordering of test cases which is turning up in the Java 7 test runs on Solaris. I ran the test 10 times without errors. I also tried running some other tests which show instability on Solaris on Java 7. Again, I am not able to reproduce these instabilities on linux on Java 7 build 132.
        Hide
        Lily Wei added a comment -

        I ran this test on Windows 7 on Java 7 build 132. I saw this same stack trace on my machine:
        4) testTriggerContainsSql(org.apache.derbyTesting.functionTests.tests.lang.Proce
        dureInTriggerTest)junit.framework.AssertionFailedError: expected:<2> but was:<4>
        at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTe
        st.checkAndResetGetConnectionProcCount(ProcedureInTriggerTest.java:514)
        at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTe
        st.testTriggerContainsSql(ProcedureInTriggerTest.java:120)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
        java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
        sorImpl.java:43)
        at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:
        112)
        at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
        at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
        at junit.extensions.TestSetup.run(TestSetup.java:23)
        at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57
        )
        I am attaching my output rjall.out here.

        Show
        Lily Wei added a comment - I ran this test on Windows 7 on Java 7 build 132. I saw this same stack trace on my machine: 4) testTriggerContainsSql(org.apache.derbyTesting.functionTests.tests.lang.Proce dureInTriggerTest)junit.framework.AssertionFailedError: expected:<2> but was:<4> at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTe st.checkAndResetGetConnectionProcCount(ProcedureInTriggerTest.java:514) at org.apache.derbyTesting.functionTests.tests.lang.ProcedureInTriggerTe st.testTriggerContainsSql(ProcedureInTriggerTest.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java: 112) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57 ) I am attaching my output rjall.out here.
        Hide
        Knut Anders Hatlen added a comment -

        Thanks, Lily and Rick, for running the tests.

        I think there are multiple issues with this test that causes problems when the test cases are reordered:

        • some tests leave rows in tables that are shared between the test cases
        • some tests don't drop the triggers they create
        • static counters are not reset between the test cases
        Show
        Knut Anders Hatlen added a comment - Thanks, Lily and Rick, for running the tests. I think there are multiple issues with this test that causes problems when the test cases are reordered: some tests leave rows in tables that are shared between the test cases some tests don't drop the triggers they create static counters are not reset between the test cases
        Hide
        Knut Anders Hatlen added a comment -

        Attaching a patch that attempts to loosen up the dependency on a particular ordering. The patch does the following:

        1) Adds a tearDown() method that truncates the test tables between each run and resets the static counters.

        2) Move an insert statement from testTriggerNoSql to testTriggerContainsSql. testTriggerNoSql doesn't use that row (inserting it is the last thing it does), so it's better that it's inserted in the test case that actually needs it.

        3) Added an insert statement in testTriggerNegative to compensate for the rows that are removed by tearDown(). It used to depend on these rows getting inserted by testModifiesSql, but now it should work even if testModifiesSql hasn't been run yet.

        4) Drop the triggers created in testTriggerNegative.

        With these changes, the test runs cleanly in an environment where the test failed almost consistently before.

        Show
        Knut Anders Hatlen added a comment - Attaching a patch that attempts to loosen up the dependency on a particular ordering. The patch does the following: 1) Adds a tearDown() method that truncates the test tables between each run and resets the static counters. 2) Move an insert statement from testTriggerNoSql to testTriggerContainsSql. testTriggerNoSql doesn't use that row (inserting it is the last thing it does), so it's better that it's inserted in the test case that actually needs it. 3) Added an insert statement in testTriggerNegative to compensate for the rows that are removed by tearDown(). It used to depend on these rows getting inserted by testModifiesSql, but now it should work even if testModifiesSql hasn't been run yet. 4) Drop the triggers created in testTriggerNegative. With these changes, the test runs cleanly in an environment where the test failed almost consistently before.
        Hide
        Knut Anders Hatlen added a comment -

        Committed revision 1079805.

        Show
        Knut Anders Hatlen added a comment - Committed revision 1079805.
        Hide
        Knut Anders Hatlen added a comment -

        There's still an intermittent failure in this test seen in the nightly runs: http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/testing/testlog/sol/1080669-suitesAll_diff.txt

        When testTriggerNegative runs early, it may alter the ordering of rows returned in some other test cases because it creates an index on T1.

        The attached patch makes testTriggerNegative drop the index in the cleanup section.

        Committed revision 1081081.

        Show
        Knut Anders Hatlen added a comment - There's still an intermittent failure in this test seen in the nightly runs: http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/testing/testlog/sol/1080669-suitesAll_diff.txt When testTriggerNegative runs early, it may alter the ordering of rows returned in some other test cases because it creates an index on T1. The attached patch makes testTriggerNegative drop the index in the cleanup section. Committed revision 1081081.

          People

          • Assignee:
            Knut Anders Hatlen
            Reporter:
            Knut Anders Hatlen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development