Derby
  1. Derby
  2. DERBY-4213

sttest needs to be adjusted to not run out of disk space

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 10.5.1.1, 10.6.1.0
    • Fix Version/s: 10.5.3.1, 10.6.1.0
    • Component/s: Test
    • Labels:
      None

      Description

      After eliminating some NullPointerExceptions for 10.5.1.1 (which I noticed with 10.5.1.0 testing but had also been present with 10.4.2.0) the sttest runs out of disk space before all threads are done.
      Before, with the NPEs, the test would run its course in a day or two.
      The test should be adjusted, with perhaps more deletes, so that it will no longer run out of disk space.

      1. DERBY-4213.diff1
        6 kB
        Myrna van Lunteren

        Activity

        Hide
        Myrna van Lunteren added a comment -

        I had hoped that recent fixes to the compress code might have taken care of this, but I ran sttest and the database still grew to like 40 G within the first day and filled the disk. Interestingly, there were no errors when the disk got full, instead, processing just stopped...

        Show
        Myrna van Lunteren added a comment - I had hoped that recent fixes to the compress code might have taken care of this, but I ran sttest and the database still grew to like 40 G within the first day and filled the disk. Interestingly, there were no errors when the disk got full, instead, processing just stopped...
        Hide
        Myrna van Lunteren added a comment -

        I took a quick look at the class system/sttest/utils/CompressTable.java, to see why this wasn't improved by the fix for the in-place compress, and it's because this class doesn't use in place compress.
        Instead, it uses the syntax:
        s.execute("alter table Datatypes compress");

        Perhaps this improves enough if we implement inplace compress in this class.

        Show
        Myrna van Lunteren added a comment - I took a quick look at the class system/sttest/utils/CompressTable.java, to see why this wasn't improved by the fix for the in-place compress, and it's because this class doesn't use in place compress. Instead, it uses the syntax: s.execute("alter table Datatypes compress"); Perhaps this improves enough if we implement inplace compress in this class.
        Hide
        Myrna van Lunteren added a comment -

        I tried to replace the section in utils/CompressTable.java which looks like it tries to compress the Datatypes table, with a call to SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE(.,.,.,.,.), but so far, I don't see the compress being called.
        However, I notice that although there's code that deletes rows being call, but only very, very few rows actually get deleted - most of the deletes find no rows to delete.
        At which point I don't think compress would do much anyway.

        I'm take a closer look at the delete rows code.

        Show
        Myrna van Lunteren added a comment - I tried to replace the section in utils/CompressTable.java which looks like it tries to compress the Datatypes table, with a call to SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE(.,.,.,.,.), but so far, I don't see the compress being called. However, I notice that although there's code that deletes rows being call, but only very, very few rows actually get deleted - most of the deletes find no rows to delete. At which point I don't think compress would do much anyway. I'm take a closer look at the delete rows code.
        Hide
        Myrna van Lunteren added a comment - - edited

        I'm attaching a patch which tries to address the growth of the database in sttest by the following:

        • adjust the compress() method with syntax that works (using SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE) and hang it in the test.
          The method Sttest.compress wasn't used before.
          As I didn't think it mattered much where this was called, but I didn't want it very often, I hung this right after a Memcheck.
        • adjust the delete code to pick rows by doing select id from Datatypes where id >= <randomnumber> and if there is no suitable row in the database, try with a different random number.
          Before, it was using the exact random number, and it was adding the resulting id value to the list, resulting in lots of attempts to delete a row with id '0'.
          Now, the likelihood of finding a row is much better, and so more rows actually get deleted.

        With this code, the database stays fairly small in my test runs (granted, I built with both targetmin and targetmax smaller).

        Some comments:

        • I also updated the utils/CompressTable with working syntax for in-place-compress, however, as far as I can tell, this class, or its methods, are not used anywhere.
        • Sometimes, (especially with smaller targetmin/max) the code that builds a list of rows to be deleted, will end up with a number of identical ids to be deleted. I figured this was ok, it's still better than hoping for a random number to exist in the database.
        • I noticed various minor inconsistencies that I don't intend to address at this time, for instance:
        • there are (still) some unused objects floating around.
        • sometimes, the message 'hit targetmax with <rowcount> ...' has a null date. I didn't look into this as it seemed unrelated to my changes.

        Review is welcome, but I plan to commit this fairly soon...If there are no comments on this patch, I'll commit it tomorrow and backport to 10.5.

        Show
        Myrna van Lunteren added a comment - - edited I'm attaching a patch which tries to address the growth of the database in sttest by the following: adjust the compress() method with syntax that works (using SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE) and hang it in the test. The method Sttest.compress wasn't used before. As I didn't think it mattered much where this was called, but I didn't want it very often, I hung this right after a Memcheck. adjust the delete code to pick rows by doing select id from Datatypes where id >= <randomnumber> and if there is no suitable row in the database, try with a different random number. Before, it was using the exact random number, and it was adding the resulting id value to the list, resulting in lots of attempts to delete a row with id '0'. Now, the likelihood of finding a row is much better, and so more rows actually get deleted. With this code, the database stays fairly small in my test runs (granted, I built with both targetmin and targetmax smaller). Some comments: I also updated the utils/CompressTable with working syntax for in-place-compress, however, as far as I can tell, this class, or its methods, are not used anywhere. Sometimes, (especially with smaller targetmin/max) the code that builds a list of rows to be deleted, will end up with a number of identical ids to be deleted. I figured this was ok, it's still better than hoping for a random number to exist in the database. I noticed various minor inconsistencies that I don't intend to address at this time, for instance: there are (still) some unused objects floating around. sometimes, the message 'hit targetmax with <rowcount> ...' has a null date. I didn't look into this as it seemed unrelated to my changes. Review is welcome, but I plan to commit this fairly soon...If there are no comments on this patch, I'll commit it tomorrow and backport to 10.5.
        Hide
        Myrna van Lunteren added a comment -

        I committed the patch to trunk (revision 806699) and backported to 10.5 (revision 806715).
        It's always possible to adjust this further if anyone has comments, but for now, I'm resolving the issue, and as I logged it, I'm closing it too.

        Show
        Myrna van Lunteren added a comment - I committed the patch to trunk (revision 806699) and backported to 10.5 (revision 806715). It's always possible to adjust this further if anyone has comments, but for now, I'm resolving the issue, and as I logged it, I'm closing it too.

          People

          • Assignee:
            Myrna van Lunteren
            Reporter:
            Myrna van Lunteren
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development