Derby
  1. Derby
  2. DERBY-5145

Provide option to limit compatibility test to combinations that include trunk

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.8.1.2
    • Fix Version/s: 10.9.1.0
    • Component/s: Test
    • Labels:
      None

      Description

      The compatibility test by default tests all possible client/server combinations. This includes testing already released versions against each other, like 10.6.2.1 client against 10.1.2.1 server. Testing for incompatibilities between already released versions isn't of much interest for the ongoing development. For example, when testing a change on trunk, you're really only interested in the combinations that have trunk either on the server side or on the client side. If we could filter out the combinations that didn't test any code on trunk (or some other branch we want to test), the time to run the tests could be significantly reduced.

      1. latestOnly.diff
        4 kB
        Knut Anders Hatlen

        Activity

        Hide
        Knut Anders Hatlen added a comment -

        This can almost be achieved with the current options.

        If you have a compatibilitytest.properties file set up with the following Derby versions

        derby.versions=18
        derby.version0=10.0.2.1
        ...
        derby.version17=Trunk

        you can limit it to only use the client from trunk by adding the line

        test.singleClient=17

        Then you'll test the client from trunk against all server versions.

        Similarly, you can test all client versions against the server from trunk by using the property test.singleServer instead of test.singleClient.

        This approach however requires you to run the compatibility tests twice, with two different derbycompatibilitytest.properties configuration files. It would have been better if we had a property that enabled you to test the same set of combinations in a single invocation.

        Show
        Knut Anders Hatlen added a comment - This can almost be achieved with the current options. If you have a compatibilitytest.properties file set up with the following Derby versions derby.versions=18 derby.version0=10.0.2.1 ... derby.version17=Trunk you can limit it to only use the client from trunk by adding the line test.singleClient=17 Then you'll test the client from trunk against all server versions. Similarly, you can test all client versions against the server from trunk by using the property test.singleServer instead of test.singleClient. This approach however requires you to run the compatibility tests twice, with two different derbycompatibilitytest.properties configuration files. It would have been better if we had a property that enabled you to test the same set of combinations in a single invocation.
        Hide
        Knut Anders Hatlen added a comment -

        Attaching a patch that adds a new flag, test.latestOnly, which can be set in compatibilitytest.properties in order to limit the testing to combinations that use the latest version either on the client side or on the server side.

        If you run the compatibility tests with all official releases (18 versions currently) + trunk, using a single JVM version, the number of combinations that are tested will drop from 361 (N²) to 37 (2N-1) when the property is set. In other words, the number of combinations grows linearly with the number of Derby versions instead of quadratically. (The number of combinations still grows quadratically with the number of JVM versions, though.)

        Show
        Knut Anders Hatlen added a comment - Attaching a patch that adds a new flag, test.latestOnly, which can be set in compatibilitytest.properties in order to limit the testing to combinations that use the latest version either on the client side or on the server side. If you run the compatibility tests with all official releases (18 versions currently) + trunk, using a single JVM version, the number of combinations that are tested will drop from 361 (N²) to 37 (2N-1) when the property is set. In other words, the number of combinations grows linearly with the number of Derby versions instead of quadratically. (The number of combinations still grows quadratically with the number of JVM versions, though.)
        Hide
        Bryan Pendleton added a comment -

        That seems quite sensible.

        Show
        Bryan Pendleton added a comment - That seems quite sensible.
        Hide
        Knut Anders Hatlen added a comment -

        Thanks, Bryan. I've committed revision 1132648 and added the flag to the compatibilitytest.properties file used by the nightly tests, so hopefully we can save some CPU cycles in the next run.

        Show
        Knut Anders Hatlen added a comment - Thanks, Bryan. I've committed revision 1132648 and added the flag to the compatibilitytest.properties file used by the nightly tests, so hopefully we can save some CPU cycles in the next run.

          People

          • Assignee:
            Knut Anders Hatlen
            Reporter:
            Knut Anders Hatlen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development