Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.5.1.1
    • Fix Version/s: 10.6.1.0
    • Component/s: Test
    • Labels:
      None

      Description

      The in-memory back end should be tested as part of the standard regression tests.

      The following is a good start:
      o simple create / boot test
      o create in-memory db, backup, restore with default storage engine, modify, restore with createFrom into memory again
      o add the functional test(s) written by Cheng
      o more unit tests

      On a longer term, we should consider to add the possibility to run all or a subset of the general regression tests with the in-memory back end.

      1. buildbreak.diff
        0.7 kB
        Knut Anders Hatlen
      2. derby-4085-1a-basic_tests.diff
        16 kB
        Kristian Waagan
      3. derby-4085-1a-basic_tests.stat
        0.4 kB
        Kristian Waagan
      4. derby-4085-1b-basic_tests.diff
        16 kB
        Kristian Waagan
      5. derby-4085-2a-mog_func_test.diff
        39 kB
        Kristian Waagan
      6. derby-4085-2a-mog_func_test.stat
        0.5 kB
        Kristian Waagan
      7. derby-4085-3a-enable_in_suitesAll.diff
        0.9 kB
        Kristian Waagan
      8. derby-4085-4a-improved_assertion_reporting.diff
        0.7 kB
        Kristian Waagan
      9. derby-4085-5a-report_seed_on_failure.diff
        3 kB
        Kristian Waagan
      10. derby-4085-6a-fixed_seed.diff
        0.8 kB
        Kristian Waagan

        Issue Links

          Activity

          Hide
          Kristian Waagan added a comment -

          Closing this issue, as the initially planned work has been completed.
          Testing can always be improved, and this work should continue outside of this Jira. Note especially the issue logged to stabilize the MogTest.

          Show
          Kristian Waagan added a comment - Closing this issue, as the initially planned work has been completed. Testing can always be improved, and this work should continue outside of this Jira. Note especially the issue logged to stabilize the MogTest.
          Hide
          Kristian Waagan added a comment -

          That's a good idea, Kathey!

          Patch 'derby-4085-6a-fixed_seed.diff' fixes the seed in an attempt to stabilize the test.
          Committed to trunk with revision 771180.
          I'll await results from the daily testing, since all I know is that the test passes with the seed when run on my own system. If it fails again, a different seed must be used.
          If the daily test passes on all platforms, I'll backport the change to the 10.5 branch.

          I have also created DERBY-4209 to track the stabilization work.

          Show
          Kristian Waagan added a comment - That's a good idea, Kathey! Patch 'derby-4085-6a-fixed_seed.diff' fixes the seed in an attempt to stabilize the test. Committed to trunk with revision 771180. I'll await results from the daily testing, since all I know is that the test passes with the seed when run on my own system. If it fails again, a different seed must be used. If the daily test passes on all platforms, I'll backport the change to the 10.5 branch. I have also created DERBY-4209 to track the stabilization work.
          Hide
          Kathey Marsden added a comment -

          Kristian asked:
          >Shall we disable the test?

          Does the test have any value if we always use a seed that we know works? If so we might be able to temporarily stabilize the test, but keep it in the suite.

          Show
          Kathey Marsden added a comment - Kristian asked: >Shall we disable the test? Does the test have any value if we always use a seed that we know works? If so we might be able to temporarily stabilize the test, but keep it in the suite.
          Hide
          Kristian Waagan added a comment -

          An example where the test fails with both the in-memory and the on-disk back end (with different seeds though):
          1) testClusMogInMemory(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: MOG configurations differ, seed=1240342031920 expected:<4> but was:<3>
          2) testClusMogOnDisk(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: MOG configurations differ, seed=1240342037948 expected:<4> but was:<3>

          Show
          Kristian Waagan added a comment - An example where the test fails with both the in-memory and the on-disk back end (with different seeds though): 1) testClusMogInMemory(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: MOG configurations differ, seed=1240342031920 expected:<4> but was:<3> 2) testClusMogOnDisk(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: MOG configurations differ, seed=1240342037948 expected:<4> but was:<3>
          Hide
          Kristian Waagan added a comment -

          I'm not sure what to do about this test - it fails in two different ways;
          a) Inconsistent number of buckets computed (by Java computation and SQL computation).
          b) Error too big.

          I don't know the test well enough to say if it is correct to increase the error threshold.
          The error regarding the buckets seems more like a real problem to me.

          Shall we disable the test?
          We can also hope that Cheng, who wrote the original test, will have some free cycles to help us out.

          Show
          Kristian Waagan added a comment - I'm not sure what to do about this test - it fails in two different ways; a) Inconsistent number of buckets computed (by Java computation and SQL computation). b) Error too big. I don't know the test well enough to say if it is correct to increase the error threshold. The error regarding the buckets seems more like a real problem to me. Shall we disable the test? We can also hope that Cheng, who wrote the original test, will have some free cycles to help us out.
          Hide
          Kathey Marsden added a comment -

          I saw the following failure with IBM 1.4.2 on trunk, Linux. Should I just post it here or open a separate issue?

          1) testClusMogInMemory(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: Error too big;6.911593155944432E-4 >= 1.0E-6, seed=1240288888777
          at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.compare(MogTest.java:159)
          at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.doTestClusMog(MogTest.java:143)
          at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.testClusMogInMemory(MogTest.java:72)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java(Compiled Code))
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)

          Show
          Kathey Marsden added a comment - I saw the following failure with IBM 1.4.2 on trunk, Linux. Should I just post it here or open a separate issue? 1) testClusMogInMemory(org.apache.derbyTesting.functionTests.tests.memorydb.MogTest)junit.framework.AssertionFailedError: Error too big;6.911593155944432E-4 >= 1.0E-6, seed=1240288888777 at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.compare(MogTest.java:159) at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.doTestClusMog(MogTest.java:143) at org.apache.derbyTesting.functionTests.tests.memorydb.MogTest.testClusMogInMemory(MogTest.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code)) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code)) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code)) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java(Compiled Code)) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          Hide
          Kristian Waagan added a comment -

          The test continues to fail for certain seeds.
          In the case I have been looking at, the Java computation ends up with 6 buckets (0 - 5), whereas the SQL computation ends up with 5 buckets (0-3, 5).
          This is in the second round of computation, the first round works fine.

          Show
          Kristian Waagan added a comment - The test continues to fail for certain seeds. In the case I have been looking at, the Java computation ends up with 6 buckets (0 - 5), whereas the SQL computation ends up with 5 buckets (0-3, 5). This is in the second round of computation, the first round works fine.
          Hide
          Kristian Waagan added a comment -

          Thank you, Cheng.

          Patch 5a makes the test report the seed when it fails in an assert. This allows us to repeat the run, as described by Cheng.
          Committed patch to trunk with revision 761204.

          Show
          Kristian Waagan added a comment - Thank you, Cheng. Patch 5a makes the test report the seed when it fails in an assert. This allows us to repeat the run, as described by Cheng. Committed patch to trunk with revision 761204.
          Hide
          Cheng Che Chen added a comment -

          The test program uses the current time to initialize its random number generator, so every run is different. In the case of a failure, you'd have to note the random seed that it printed out at the beginning of the run, and then modify one line of the program to reuse that seed instead of using the current time, in order to repeat the run.

          Show
          Cheng Che Chen added a comment - The test program uses the current time to initialize its random number generator, so every run is different. In the case of a failure, you'd have to note the random seed that it printed out at the beginning of the run, and then modify one line of the program to reuse that seed instead of using the current time, in order to repeat the run.
          Hide
          Kristian Waagan added a comment -

          Committed patch 4a to trunk with revision 760380.

          The test has failed more than once in the daily tests. I added the improved error reporting to see if the error tolerance level is too low, or if there is something else wrong with the test.

          Show
          Kristian Waagan added a comment - Committed patch 4a to trunk with revision 760380. The test has failed more than once in the daily tests. I added the improved error reporting to see if the error tolerance level is too low, or if there is something else wrong with the test.
          Hide
          Kristian Waagan added a comment -

          The MOG test failed on a SPARC machine (daily testing). I'll check if it happens again, and investigate.

          Show
          Kristian Waagan added a comment - The MOG test failed on a SPARC machine (daily testing). I'll check if it happens again, and investigate.
          Hide
          Kristian Waagan added a comment -

          Patch 3a enables the in-memory back end tests in suites.All.
          This will give us more confidence that the in-memory back end works on more platforms.

          Committed to trunk with revision 759211.

          Show
          Kristian Waagan added a comment - Patch 3a enables the in-memory back end tests in suites.All. This will give us more confidence that the in-memory back end works on more platforms. Committed to trunk with revision 759211.
          Hide
          Kristian Waagan added a comment -

          Committed patch 2a to trunk with revision 759181.
          I will also enable the tests as part of suites.All to get exposure on more platforms.

          Show
          Kristian Waagan added a comment - Committed patch 2a to trunk with revision 759181. I will also enable the tests as part of suites.All to get exposure on more platforms.
          Hide
          Kristian Waagan added a comment -

          Patch 2a adds the functional test contributed by Cheng Che Chen. I've made some changes to integrate the test into Derby's test framework.
          I see rather large variations in the time it takes to run the test (everything from less then a second to more then one minute), could it be due to the use of random?

          Change _Suite to return an empty Suite if JSR 169 / JavaME is being used.

          Patch ready for review.

          Show
          Kristian Waagan added a comment - Patch 2a adds the functional test contributed by Cheng Che Chen. I've made some changes to integrate the test into Derby's test framework. I see rather large variations in the time it takes to run the test (everything from less then a second to more then one minute), could it be due to the use of random? Change _Suite to return an empty Suite if JSR 169 / JavaME is being used. Patch ready for review.
          Hide
          Knut Anders Hatlen added a comment -

          Here's a patch that fixes the build break. Since BasicInMemoryDbTest uses DriverManager, it must be compiled against java14compile.classpath. The problem is only seen if jsr169compile.classpath points to JSR-169 + Foundation libraries.

          Committed revision 753279.

          When this test is enabled as part of a suite, we should make sure it's not run on Java ME. Or we should change the test so that it doesn't need DriverManager, in which case it should run on Java ME too, and the change that I just committed should be reverted.

          Show
          Knut Anders Hatlen added a comment - Here's a patch that fixes the build break. Since BasicInMemoryDbTest uses DriverManager, it must be compiled against java14compile.classpath. The problem is only seen if jsr169compile.classpath points to JSR-169 + Foundation libraries. Committed revision 753279. When this test is enabled as part of a suite, we should make sure it's not run on Java ME. Or we should change the test so that it doesn't need DriverManager, in which case it should run on Java ME too, and the change that I just committed should be reverted.
          Hide
          Knut Anders Hatlen added a comment -

          I see this build-failure in my environment:

          memorytests:
          [javac] Compiling 6 source files to /code/derby/trunk0/classes

          compile:
          [javac] Compiling 2 source files to /code/derby/trunk0/classes
          [javac] /code/derby/trunk0/java/testing/org/apache/derbyTesting/functionTests/tests/memorydb/BasicInMemoryDbTest.java:28: cannot find symbol
          [javac] symbol : class DriverManager
          [javac] location: package java.sql
          [javac] import java.sql.DriverManager;
          [javac] ^
          [javac] /code/derby/trunk0/java/testing/org/apache/derbyTesting/functionTests/tests/memorydb/BasicInMemoryDbTest.java:61: cannot find symbol

          Show
          Knut Anders Hatlen added a comment - I see this build-failure in my environment: memorytests: [javac] Compiling 6 source files to /code/derby/trunk0/classes compile: [javac] Compiling 2 source files to /code/derby/trunk0/classes [javac] /code/derby/trunk0/java/testing/org/apache/derbyTesting/functionTests/tests/memorydb/BasicInMemoryDbTest.java:28: cannot find symbol [javac] symbol : class DriverManager [javac] location: package java.sql [javac] import java.sql.DriverManager; [javac] ^ [javac] /code/derby/trunk0/java/testing/org/apache/derbyTesting/functionTests/tests/memorydb/BasicInMemoryDbTest.java:61: cannot find symbol
          Hide
          Kristian Waagan added a comment -

          Committed patch 1b to trunk with revision 753210.
          The difference from revision a is the changed subSubProtocol.

          Note that the tests must be run manually for the time being.

          Show
          Kristian Waagan added a comment - Committed patch 1b to trunk with revision 753210. The difference from revision a is the changed subSubProtocol. Note that the tests must be run manually for the time being.
          Hide
          Kristian Waagan added a comment -

          Patch 1a adds a very simple boot test and the backup / modify / restore test. The tests haven't been added to suites.All or the ant targets yet.
          Patch ready for review.

          I'm not sure if we should continue to connect to the databases manually, or if it should be possible to add a decorator or set an option to use a specific subSubProtocol.

          Show
          Kristian Waagan added a comment - Patch 1a adds a very simple boot test and the backup / modify / restore test. The tests haven't been added to suites.All or the ant targets yet. Patch ready for review. I'm not sure if we should continue to connect to the databases manually, or if it should be possible to add a decorator or set an option to use a specific subSubProtocol.

            People

            • Assignee:
              Kristian Waagan
              Reporter:
              Kristian Waagan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development