Derby
  1. Derby
  2. DERBY-5638

intermittent test failure in test_05_ClobNegative when running full largedata._Suite; LobLimitsTestjava.sql.SQLException: Table/View 'BLOBTBL' already exists in Schema 'APP'.

    Details

    • Urgency:
      Normal
    • Bug behavior facts:
      Regression Test Failure

      Description

      I've seen the following failure when running the largedata suite:

      (emb)largedata.Derby5624Test.testDERBY_5624 used 518403 ms .
      (emb)largedata.LobLimitsTest.test_01_Blob used 2422 ms .
      (emb)largedata.LobLimitsTest.test_02_BlobNegative used 31 ms .
      (emb)largedata.LobLimitsTest.test_03_Clob1 used 2375 ms .
      (emb)largedata.LobLimitsTest.test_04_Clob2 used 3234 ms .
      (emb)largedata.LobLimitsTest.test_05_ClobNegative used 516 ms .
      (net)largedata.LobLimitsTest.test_01_Blob used 5360 ms .
      (net)largedata.LobLimitsTest.test_02_BlobNegative used 32 ms .
      (net)largedata.LobLimitsTest.test_03_Clob1 used 2078 ms .
      (net)largedata.LobLimitsTest.test_04_Clob2 used 2390 ms .
      (net)largedata.LobLimitsTest.test_05_ClobNegative used 938 ms .
      (emb)largedata.LobLimitsTest.test_01_Blob used 9188238 ms .
      (emb)largedata.LobLimitsTest.test_02_BlobNegative used 109 ms .
      (emb)largedata.LobLimitsTest.test_03_Clob1 used 8116714 ms .
      (emb)largedata.LobLimitsTest.test_04_Clob2 used 2164002 ms .
      (emb)largedata.LobLimitsTest.test_05_ClobNegative used 685745 ms E
      Time: 22,320.138
      There was 1 error:
      1) LobLimitsTestjava.sql.SQLException: Table/View 'BLOBTBL' already exists in Schema 'APP'.
      at org.apache.derby.client.am.SQLExceptionFactory40.getSQLException(Unknown Source)
      at org.apache.derby.client.am.SqlException.getSQLException(Unknown Source)
      at org.apache.derby.client.am.Statement.execute(Unknown Source)
      at org.apache.derbyTesting.functionTests.tests.largedata.LobLimitsTest.setupTables(LobLimitsTest.java:107)
      at org.apache.derbyTesting.functionTests.tests.largedata.LobLimitsTest$1.decorateSQL(LobLimitsTest.java:141)
      at org.apache.derbyTesting.junit.CleanDatabaseTestSetup.setUp(CleanDatabaseTestSetup.java:112)
      at junit.extensions.TestSetup$1.protect(TestSetup.java:20)
      at junit.extensions.TestSetup.run(TestSetup.java:25)
      at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
      at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
      at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
      at junit.extensions.TestSetup.run(TestSetup.java:25)
      at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
      at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
      at junit.extensions.TestSetup.run(TestSetup.java:25)
      at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
      at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
      at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
      at junit.extensions.TestSetup.run(TestSetup.java:25)
      Caused by: org.apache.derby.client.am.SqlException: Table/View 'BLOBTBL' already exists in Schema 'APP'.
      at org.apache.derby.client.am.Statement.completeSqlca(Unknown Source)
      at org.apache.derby.client.am.Statement.completeExecuteImmediate(Unknown Source)
      at org.apache.derby.client.net.NetStatementReply.parseEXCSQLIMMreply(Unknown Source)
      at org.apache.derby.client.net.NetStatementReply.readExecuteImmediate(Unknown Source)
      at org.apache.derby.client.net.StatementReply.readExecuteImmediate(Unknown Source)
      at org.apache.derby.client.net.NetStatement.readExecuteImmediate_(Unknown Source)
      at org.apache.derby.client.am.Statement.readExecuteImmediate(Unknown Source)
      at org.apache.derby.client.am.Statement.flowExecute(Unknown Source)
      at org.apache.derby.client.am.Statement.executeX(Unknown Source)
      ... 26 more

      Unfortunately, when this happens, there seems to be no 'fail' directory created. The derby.log in the system directory looks very innocent (just some start up and shutting down of the database), and the serverConsoleOutput.log only has the typical 'failed to find db 'wombat' messages'.

      Note, when this happens, the suite exits, so that instead of the expected 20 (or 21 on windows, see DERBY-5624 for reason for skipping on Linux default installs with 1024 max open files) we only get 15 (or 16) tests run - if the test doesn't fail it goes on to run the last 5 fixtures again for network server.

      1. runallWithPrintlnTrace.out
        19 kB
        Mamta A. Satoor
      2. runallForSuccessfulLargeDataRun.out
        1 kB
        Mamta A. Satoor
      3. derbyWithRollbackInTest_05.log
        178 kB
        Mamta A. Satoor
      4. derbyPass.log
        188 kB
        Mamta A. Satoor
      5. derbyFailed.log
        164 kB
        Mamta A. Satoor
      6. derbyallFailWithPrintlnTrace.log
        164 kB
        Mamta A. Satoor
      7. DERBY5638_patch2_diff.txt
        3 kB
        Mamta A. Satoor
      8. DERBY5638_patch1_diff.txt
        1 kB
        Mamta A. Satoor
      9. CompleteRunallWithPrintlnTrace.out
        26 kB
        Mamta A. Satoor

        Issue Links

          Activity

          Hide
          Knut Anders Hatlen added a comment -

          [bulk update: close all resolved issues that haven't had any activity the last year]

          Show
          Knut Anders Hatlen added a comment - [bulk update: close all resolved issues that haven't had any activity the last year]
          Hide
          Myrna van Lunteren added a comment -

          I'm marking this as backport_reject_10_7 because the test was not in junit format with 10.7, so this failure would not occur.

          Show
          Myrna van Lunteren added a comment - I'm marking this as backport_reject_10_7 because the test was not in junit format with 10.7, so this failure would not occur.
          Hide
          Myrna van Lunteren added a comment -

          backported to 10.8 with revision 1384589 (http://svn.apache.org/viewvc?view=revision&revision=1384589) from trunk. largedata._Suite ran fine on windows 7 with ibm 1.6.

          Show
          Myrna van Lunteren added a comment - backported to 10.8 with revision 1384589 ( http://svn.apache.org/viewvc?view=revision&revision=1384589 ) from trunk. largedata._Suite ran fine on windows 7 with ibm 1.6.
          Hide
          Myrna van Lunteren added a comment -

          assigning to me for the backport.

          Show
          Myrna van Lunteren added a comment - assigning to me for the backport.
          Hide
          Kathey Marsden added a comment -

          Reopen for backport. Assign yourself temporarily to backport and reassign to Mamta before resolving

          Show
          Kathey Marsden added a comment - Reopen for backport. Assign yourself temporarily to backport and reassign to Mamta before resolving
          Hide
          Mamta A. Satoor added a comment -

          Changes were committed for this jira in March.

          Show
          Mamta A. Satoor added a comment - Changes were committed for this jira in March.
          Hide
          Mamta A. Satoor added a comment -

          The changes committed so far for this jira will be ok for backport if we decide to do that at some point.

          Show
          Mamta A. Satoor added a comment - The changes committed so far for this jira will be ok for backport if we decide to do that at some point.
          Hide
          Mamta A. Satoor added a comment -

          The large data test ran into out of disk space problem in one out of the 2 runs since the checkin on the March 15th. After clearing up the space by deleting few old runs, the test ran fine the 2nd time.

          It will be good idea to reduce the size of the wombat database after every large data suite has finished thus successful runs will not occupy too much disk space. I went with Knut's recommendation to add truncate table as part of the cleanup in deleteTable() method. I kept the delete from table code too because delete follows a different code path than truncate and it will be good for us to test both code paths in case of large data testing. The deleteTable() method looks as follows in my codeline
          private void deleteTable(String table,
          int expectedRows) throws SQLException

          { Statement s = createStatement(); //Keep the delete call to exercise delete of long blobs and clobs. // This is a separate code path through Derby compared to truncate // table code path. int count = s.executeUpdate( "DELETE FROM " + JDBC.escape(table)); //Adding truncate call which will give back the disk space being // used by the table at commit time rather than wait for test // infrastructure to drop the table. s.executeUpdate("TRUNCATE TABLE " + JDBC.escape(table)); commit(); verifyTest(count, expectedRows, "Rows deleted ="); }

          What I found with this change is that the wombat/log directory is much larger than it was without the truncate table call.

          log directory looks as follows with the svn codeline(ie without the truncate table call)
          $ ls -l log
          total 3702
          ----------+ 1 mamta mkgroup 48 Mar 14 16:56 log.ctrl
          ----------+ 1 mamta mkgroup 2738408 Mar 14 16:56 log440.dat
          ----------+ 1 mamta mkgroup 1048576 Mar 14 19:00 log441.dat
          ----------+ 1 mamta mkgroup 48 Mar 14 16:56 logmirror.ctrl

          log directory with large data suite run with truncate table call as shown in the deleteTable method looks as follows
          $ ls -l log
          total 312254
          ----------+ 1 mamta mkgroup 48 Mar 19 17:02 log.ctrl
          ----------+ 1 mamta mkgroup 26765701 Mar 19 17:00 log416.dat
          ----------+ 1 mamta mkgroup 33559814 Mar 19 17:00 log417.dat
          ----------+ 1 mamta mkgroup 33560037 Mar 19 17:01 log418.dat
          ----------+ 1 mamta mkgroup 33791143 Mar 19 17:01 log419.dat
          ----------+ 1 mamta mkgroup 31539171 Mar 19 17:01 log420.dat
          ----------+ 1 mamta mkgroup 6268114 Mar 19 17:01 log421.dat
          ----------+ 1 mamta mkgroup 3071813 Mar 19 17:01 log422.dat
          ----------+ 1 mamta mkgroup 31544805 Mar 19 17:01 log423.dat
          ----------+ 1 mamta mkgroup 33529672 Mar 19 17:01 log424.dat
          ----------+ 1 mamta mkgroup 33503893 Mar 19 17:01 log425.dat
          ----------+ 1 mamta mkgroup 33768332 Mar 19 17:02 log426.dat
          ----------+ 1 mamta mkgroup 17774482 Mar 19 17:02 log427.dat
          ----------+ 1 mamta mkgroup 1048576 Mar 19 18:47 log428.dat
          ----------+ 1 mamta mkgroup 48 Mar 19 17:02 logmirror.ctrl

          I talked to Mike about the change in number of files in log directory and he mentioned that if we actually look at the number in the log file names(for eg for428.dat), we will see that Derby didn't have to write as much to the log when we use truncate table which is a good sign. As can be seen above, with truncate table, the last log files name is log428.dat whereas log file name with trunk codeline run is log441.dat

          The reason we see large number of log files with truncate table is that probably a checkpoint was not issued. Checkpoints are issued periodically and among other things, it;s frequency also depends on how many log files have been written so far.

          I tried connecting to the wombat directory with large number of log files through ij and after getting out of ij, I saw that there were only 3 log files left in log directory as shown below
          $ ls -l
          total 1026
          ----------+ 1 mamta mkgroup 48 Mar 20 10:20 log.ctrl
          ----------+ 1 mamta mkgroup 1048576 Mar 20 10:20 log428.dat
          ----------+ 1 mamta mkgroup 48 Mar 20 10:20 logmirror.ctrl

          The reason for reduction in number of log files is as part of booting up wombat through ij, we applied all the left over log files to the database and hence as part of recovery, the log files were deleted since they are not needed any more.

          Based on this experiment, in addition to having truncate table in deleteTable() method, I plan on adding a test fixture to LobLimitsTest class and that test fixture will just shutodwn the databases. As part of the shutdown, the logs will get applied and hence at the end of the suite, the size of the wombat database will be much smaller and there won't be left over huge log files left behind after a successful run.

          Once I have the tests running successfully with this change and verify that log directory is indeed much smaller, i will post a patch.

          Show
          Mamta A. Satoor added a comment - The large data test ran into out of disk space problem in one out of the 2 runs since the checkin on the March 15th. After clearing up the space by deleting few old runs, the test ran fine the 2nd time. It will be good idea to reduce the size of the wombat database after every large data suite has finished thus successful runs will not occupy too much disk space. I went with Knut's recommendation to add truncate table as part of the cleanup in deleteTable() method. I kept the delete from table code too because delete follows a different code path than truncate and it will be good for us to test both code paths in case of large data testing. The deleteTable() method looks as follows in my codeline private void deleteTable(String table, int expectedRows) throws SQLException { Statement s = createStatement(); //Keep the delete call to exercise delete of long blobs and clobs. // This is a separate code path through Derby compared to truncate // table code path. int count = s.executeUpdate( "DELETE FROM " + JDBC.escape(table)); //Adding truncate call which will give back the disk space being // used by the table at commit time rather than wait for test // infrastructure to drop the table. s.executeUpdate("TRUNCATE TABLE " + JDBC.escape(table)); commit(); verifyTest(count, expectedRows, "Rows deleted ="); } What I found with this change is that the wombat/log directory is much larger than it was without the truncate table call. log directory looks as follows with the svn codeline(ie without the truncate table call) $ ls -l log total 3702 ----------+ 1 mamta mkgroup 48 Mar 14 16:56 log.ctrl ----------+ 1 mamta mkgroup 2738408 Mar 14 16:56 log440.dat ----------+ 1 mamta mkgroup 1048576 Mar 14 19:00 log441.dat ----------+ 1 mamta mkgroup 48 Mar 14 16:56 logmirror.ctrl log directory with large data suite run with truncate table call as shown in the deleteTable method looks as follows $ ls -l log total 312254 ----------+ 1 mamta mkgroup 48 Mar 19 17:02 log.ctrl ----------+ 1 mamta mkgroup 26765701 Mar 19 17:00 log416.dat ----------+ 1 mamta mkgroup 33559814 Mar 19 17:00 log417.dat ----------+ 1 mamta mkgroup 33560037 Mar 19 17:01 log418.dat ----------+ 1 mamta mkgroup 33791143 Mar 19 17:01 log419.dat ----------+ 1 mamta mkgroup 31539171 Mar 19 17:01 log420.dat ----------+ 1 mamta mkgroup 6268114 Mar 19 17:01 log421.dat ----------+ 1 mamta mkgroup 3071813 Mar 19 17:01 log422.dat ----------+ 1 mamta mkgroup 31544805 Mar 19 17:01 log423.dat ----------+ 1 mamta mkgroup 33529672 Mar 19 17:01 log424.dat ----------+ 1 mamta mkgroup 33503893 Mar 19 17:01 log425.dat ----------+ 1 mamta mkgroup 33768332 Mar 19 17:02 log426.dat ----------+ 1 mamta mkgroup 17774482 Mar 19 17:02 log427.dat ----------+ 1 mamta mkgroup 1048576 Mar 19 18:47 log428.dat ----------+ 1 mamta mkgroup 48 Mar 19 17:02 logmirror.ctrl I talked to Mike about the change in number of files in log directory and he mentioned that if we actually look at the number in the log file names(for eg for428.dat), we will see that Derby didn't have to write as much to the log when we use truncate table which is a good sign. As can be seen above, with truncate table, the last log files name is log428.dat whereas log file name with trunk codeline run is log441.dat The reason we see large number of log files with truncate table is that probably a checkpoint was not issued. Checkpoints are issued periodically and among other things, it;s frequency also depends on how many log files have been written so far. I tried connecting to the wombat directory with large number of log files through ij and after getting out of ij, I saw that there were only 3 log files left in log directory as shown below $ ls -l total 1026 ----------+ 1 mamta mkgroup 48 Mar 20 10:20 log.ctrl ----------+ 1 mamta mkgroup 1048576 Mar 20 10:20 log428.dat ----------+ 1 mamta mkgroup 48 Mar 20 10:20 logmirror.ctrl The reason for reduction in number of log files is as part of booting up wombat through ij, we applied all the left over log files to the database and hence as part of recovery, the log files were deleted since they are not needed any more. Based on this experiment, in addition to having truncate table in deleteTable() method, I plan on adding a test fixture to LobLimitsTest class and that test fixture will just shutodwn the databases. As part of the shutdown, the logs will get applied and hence at the end of the suite, the size of the wombat database will be much smaller and there won't be left over huge log files left behind after a successful run. Once I have the tests running successfully with this change and verify that log directory is indeed much smaller, i will post a patch.
          Hide
          Mamta A. Satoor added a comment -

          Hi Knut, I appreciate you taking the time to look at this.

          I have another alternative approach(which requires just one line change to large data suite network server run) to fix the problem with intermittent lock time out issues possibly because of background threads trying to finish post-commit changes related to large objects.

          The issue has been that the CleanDatabaseTestSetup , after the last test fixture of embedded suite is done, tries to drop the tables but runs into lock timeout errors and hence it never finishes dropping the tables. None of these errors get reported anywhere by CleanDatabaseTestSetup and we simply move on to the next suite which in our case is network server running the large data tests. As part of CleanDatabaseTestSetup decorator for network server, we try to drop the existing objects in the database again before creating the new objects required by the new suite and the drop tables again run into lock timeouts.

          What I am suggesting as a solution is to have the network server suite use a brand new database to do it's testing via singleUseDatabaseDecorator and that way it will not run into locks held on the database used by the embedded run of large data tests. The change is fairly easy(just one line change) in LobLimitsClientTest.suite(). The changed LobLimitsClientTest,suite looks as follows
          public static Test suite()

          { return TestConfiguration.singleUseDatabaseDecorator( TestConfiguration.clientServerDecorator(LobLimitsTest.suite())); }

          The original LobLimitsClientTest,suite() in svn looks as follows
          public static Test suite()

          { return TestConfiguration.clientServerDecorator(LobLimitsTest.suite()); }

          }

          I ran the large data suite twice on my machine with this change and I don't see the lock timeout from network server in the log file anymore. I will go ahead and commit this change.

          Additionally, I will create a new jira for CleanDatabaseTestSetup about not reporting failures while trying to clean the database by dropping the objects., Either, it should report those errors in fail directory or somewhere else so we can diagnose more easily if the subsequent suite fails because of left over database objects from the previous suite. Or it can try to drop the objects as it does but if it can't drop them successfully, then it should drop the database and create a brand new database which is clean for the next suite to use. One of the things that CleanDatabaseTestSetup can do is what Knut suggested. Do TRUNCATE TABLE if it is having trouble dropping the tables and then try to drop the tables again or wait for post-commit to finish by calling T_Access.waitForPostCommitToFinish() before dropping objects from the database.

          Show
          Mamta A. Satoor added a comment - Hi Knut, I appreciate you taking the time to look at this. I have another alternative approach(which requires just one line change to large data suite network server run) to fix the problem with intermittent lock time out issues possibly because of background threads trying to finish post-commit changes related to large objects. The issue has been that the CleanDatabaseTestSetup , after the last test fixture of embedded suite is done, tries to drop the tables but runs into lock timeout errors and hence it never finishes dropping the tables. None of these errors get reported anywhere by CleanDatabaseTestSetup and we simply move on to the next suite which in our case is network server running the large data tests. As part of CleanDatabaseTestSetup decorator for network server, we try to drop the existing objects in the database again before creating the new objects required by the new suite and the drop tables again run into lock timeouts. What I am suggesting as a solution is to have the network server suite use a brand new database to do it's testing via singleUseDatabaseDecorator and that way it will not run into locks held on the database used by the embedded run of large data tests. The change is fairly easy(just one line change) in LobLimitsClientTest.suite(). The changed LobLimitsClientTest,suite looks as follows public static Test suite() { return TestConfiguration.singleUseDatabaseDecorator( TestConfiguration.clientServerDecorator(LobLimitsTest.suite())); } The original LobLimitsClientTest,suite() in svn looks as follows public static Test suite() { return TestConfiguration.clientServerDecorator(LobLimitsTest.suite()); } } I ran the large data suite twice on my machine with this change and I don't see the lock timeout from network server in the log file anymore. I will go ahead and commit this change. Additionally, I will create a new jira for CleanDatabaseTestSetup about not reporting failures while trying to clean the database by dropping the objects., Either, it should report those errors in fail directory or somewhere else so we can diagnose more easily if the subsequent suite fails because of left over database objects from the previous suite. Or it can try to drop the objects as it does but if it can't drop them successfully, then it should drop the database and create a brand new database which is clean for the next suite to use. One of the things that CleanDatabaseTestSetup can do is what Knut suggested. Do TRUNCATE TABLE if it is having trouble dropping the tables and then try to drop the tables again or wait for post-commit to finish by calling T_Access.waitForPostCommitToFinish() before dropping objects from the database.
          Hide
          Knut Anders Hatlen added a comment -

          If there's an index on the table whose rows are deleted, there will be post-commit work scheduled that could hold locks when tearDown() is executed. If that's what's causing the timeouts, you might try one of these approaches:

          1) Wait for post-commit to finish after having deleted the rows. This can be done by calling T_Access.waitForPostCommitToFinish() in a stored procedure after deleteTable(). See ReleaseCompileLocksTest for an example.

          2) Use TRUNCATE TABLE instead of DELETE in the deleteTable() method. In addition to being faster on large tables, it also avoids post-commit work and shouldn't cause any concurrent locking to happen during tearDown().

          Show
          Knut Anders Hatlen added a comment - If there's an index on the table whose rows are deleted, there will be post-commit work scheduled that could hold locks when tearDown() is executed. If that's what's causing the timeouts, you might try one of these approaches: 1) Wait for post-commit to finish after having deleted the rows. This can be done by calling T_Access.waitForPostCommitToFinish() in a stored procedure after deleteTable(). See ReleaseCompileLocksTest for an example. 2) Use TRUNCATE TABLE instead of DELETE in the deleteTable() method. In addition to being faster on large tables, it also avoids post-commit work and shouldn't cause any concurrent locking to happen during tearDown().
          Hide
          Mamta A. Satoor added a comment -

          Hi Myrna, thanks so much for taking the time to respond.

          I recall looking through the code for missing incomplete transactions and deleteTable was one of the methods I checked and I found that it has a commit of it's own as shown below
          private void deleteTable(String table,
          int expectedRows) throws SQLException

          { int count = createStatement().executeUpdate( "DELETE FROM " + JDBC.escape(table)); commit(); verifyTest(count, expectedRows, "Rows deleted ="); }

          As for singleUseDatabaseDecorator, I have been playing with it this morning. I ran the test with that fixture but I am running into test table creation/drop off problems because setup and teardown methods get called for every test fixture. But this test has test fixtures which rely on data changes from previous fixtures(the suite has been coded to run the test fixtures in specific order through TestConfiguration.orderedSuite(LobLimitsTest.class)) and hence we do not want tables to be dropped and recreated through setup and teardown around every test fixture, I am looking at tweaking the setup code to not always create the tables. Will post once I have the changes working or if I have further questions.

          Show
          Mamta A. Satoor added a comment - Hi Myrna, thanks so much for taking the time to respond. I recall looking through the code for missing incomplete transactions and deleteTable was one of the methods I checked and I found that it has a commit of it's own as shown below private void deleteTable(String table, int expectedRows) throws SQLException { int count = createStatement().executeUpdate( "DELETE FROM " + JDBC.escape(table)); commit(); verifyTest(count, expectedRows, "Rows deleted ="); } As for singleUseDatabaseDecorator, I have been playing with it this morning. I ran the test with that fixture but I am running into test table creation/drop off problems because setup and teardown methods get called for every test fixture. But this test has test fixtures which rely on data changes from previous fixtures(the suite has been coded to run the test fixtures in specific order through TestConfiguration.orderedSuite(LobLimitsTest.class)) and hence we do not want tables to be dropped and recreated through setup and teardown around every test fixture, I am looking at tweaking the setup code to not always create the tables. Will post once I have the changes working or if I have further questions.
          Hide
          Kathey Marsden added a comment -

          Hi Mamta,

          If the goal is to drop the database, I think the best thing is to use the singleUseDatabaseDecorator in TestConfiguration

          However, I think it might be better to understand why the fixtures in the test are holding locks and fix that. Probably just putting a commit at the end of each fixture will take care of that. For example, I see in test_01_Blob() that it has autocommit off the last thing it does is:

          commit();

          deleteTable("BLOBTBL", 3);

          but the deleteTable would not get committed and would hold locks. An additional commit() should clear that up.

          I have only read your last comment so forgive me if this has been covered as an option.

          Show
          Kathey Marsden added a comment - Hi Mamta, If the goal is to drop the database, I think the best thing is to use the singleUseDatabaseDecorator in TestConfiguration However, I think it might be better to understand why the fixtures in the test are holding locks and fix that. Probably just putting a commit at the end of each fixture will take care of that. For example, I see in test_01_Blob() that it has autocommit off the last thing it does is: commit(); deleteTable("BLOBTBL", 3); but the deleteTable would not get committed and would hold locks. An additional commit() should clear that up. I have only read your last comment so forgive me if this has been covered as an option.
          Hide
          Mamta A. Satoor added a comment -

          Based on the fact that all the test fixtures fnish with no problems and it is the teardiwn/setup that is running into locking issues, I will work on the suite not relying on CleanDatabaseTestSetup. Instead, use a decorator which will drop the database and recreate it with the tables needed to run the suite. Currently, the sql to create the tables is in decorateSQL() as required by CleanDatabaseTestSetup
          Test suite = new CleanDatabaseTestSetup(
          TestConfiguration.orderedSuite(LobLimitsTest.class)) {
          protected void decorateSQL(Statement s)
          throws SQLException

          { setupTables(s, biggestSize, bigSize); }

          };

          I am wondering if I have to use DropDatabaseSetup to accomplish dropping the database and then putting the setupTables in some special method so the tables get created at the beginning of the next suite. If anyone has any experience in this area, please let me know. Thanks

          Show
          Mamta A. Satoor added a comment - Based on the fact that all the test fixtures fnish with no problems and it is the teardiwn/setup that is running into locking issues, I will work on the suite not relying on CleanDatabaseTestSetup. Instead, use a decorator which will drop the database and recreate it with the tables needed to run the suite. Currently, the sql to create the tables is in decorateSQL() as required by CleanDatabaseTestSetup Test suite = new CleanDatabaseTestSetup( TestConfiguration.orderedSuite(LobLimitsTest.class)) { protected void decorateSQL(Statement s) throws SQLException { setupTables(s, biggestSize, bigSize); } }; I am wondering if I have to use DropDatabaseSetup to accomplish dropping the database and then putting the setupTables in some special method so the tables get created at the beginning of the next suite. If anyone has any experience in this area, please let me know. Thanks
          Hide
          Mamta A. Satoor added a comment -

          One of the tests run as part of large data suite is Derby5624Test which has following in protected static Test baseSuite(String name)
          return new CleanDatabaseTestSetup(
          DatabasePropertyTestSetup.setLockTimeouts(suite, 2, 4))
          {

          Mike said the test really doesn't need that lock timeout setting and may be it is interferring with the locks timeouts that we see later on in subsequent suites in large data suite. I will remove that lock timeout setting from Derby5624Test and run the test on my machine to make sure it runs. If it does, then I will go ahead and commit that change to see if the intermittent problems disappear.

          Show
          Mamta A. Satoor added a comment - One of the tests run as part of large data suite is Derby5624Test which has following in protected static Test baseSuite(String name) return new CleanDatabaseTestSetup( DatabasePropertyTestSetup.setLockTimeouts(suite, 2, 4)) { Mike said the test really doesn't need that lock timeout setting and may be it is interferring with the locks timeouts that we see later on in subsequent suites in large data suite. I will remove that lock timeout setting from Derby5624Test and run the test on my machine to make sure it runs. If it does, then I will go ahead and commit that change to see if the intermittent problems disappear.
          Hide
          Mamta A. Satoor added a comment -

          I debugged more to see why we get 5 lock timeouts with embedded driver and then 3 locks with netwrok server before running into intermittent failure. My initial thought was that we will get 5 lock time outs both with embedded and network server before we give up trying to drop the tables.

          Here is the general flow chart from dropping tables point of view when the suite is using CleanDatabaseTestSetup.

          At the end of the last test fixture in the suite, CleanDatabaseTestSetup.tearDown gets called which does following(focusing on drop table related code, ignoring the code related to dropping aliases, views etc)
          CleanDatabaseTestSetup.tearDown->
          CleanDatabaseTestSetup.cleanDatabase->
          CleanDatabaseTestSetup.removeObjects->
          JDBC.dropSchema->
          ....
          // Tables
          rs = dmd.getTables((String) null, schema, (String) null,
          GET_TABLES_TABLE);

          dropUsingDMD(s, rs, schema, "TABLE_NAME", "TABLE");
          ....
          // At this point there may be tables left due to
          // foreign key constraints leading to a dependency loop.
          // Drop any constraints that remain and then drop the tables.
          // If there are no tables then this should be a quick no-op.
          .....
          // Tables (again)
          rs = dmd.getTables((String) null, schema, (String) null,
          GET_TABLES_TABLE);
          dropUsingDMD(s, rs, schema, "TABLE_NAME", "TABLE");

          The code in dropUsingDMD is as follows
          First try to drop the objects in a batch rather than individually.If there is any BatchUpdateException while dropping the objects as a batch, drop the remaining objects inidividually in a for loop. If the for loop gets
          an exception but some of the objects before the exception got dropped successfully, then go back into the for loop and try to drop remaining objects again. This continues until either
          a)All objects are dropped and we can return
          b)no objects can be dropped because of exception

          After this CleanDatabaseTestSetup.tearDown is finished(with or without success), the control goes to the CleanDatabaseTestSetup.setUp for the next suite. Among other things, CleanDatabaseTestSetup.setUp does following 2 main steps
          1)call CleanDatabaseTestSetup.cleanDatabase (drop any objects in the db)
          2)call decorateSQL(create new objects in the db)

          Now, let's go through this code in our intermittent failure case. The last test from the embedded suite LobLimitsTest finishes with no errors and we go to CleanDatabaseTestSetup.tearDown(). This eventually leads to JDBC.dropUsingDMD as shown above.

          In JDBC.dropUsingDMD(), we first try to drop all the tables as a batch rather than individual drops. Here, we get the first lock timeout. Since we failed here, next we go in a loop to drop the tables individually. Here we manage to drop "APP"."CLOBTBL2" but get lock time out(2nd one) for trying to drop "APP"."CLOBTBL". But since the loop was able to drop at least one table, we go back into the loop to try to drop remaining tables. But that results into 3rd lock timeout for "APP"."CLOBTBL". After these tries, the control goes back to JDBC.dropSchema where we try to drop the tables yet again after dropping any constraints which may be preventing the table drops(in our case, I don't believe we have any constraints in the db). This 2nd attempt at dropping the table takes us back to JDBC.dropUsingDMD.Again we try to drop all the tables as batch but get the 4th lock timeout for "APP"."CLOBTBL". Next, we try to drop the tables inidividually in a loop, but the drop of "APP"."CLOBTBL" fails with 5th lock timeout. And since we were not able to drop any table in the loop, we exit out of the loop and eventually exit out of CleanDatabaseTestSetup.tearDown without reporting all this as a failure even though we were not able to clean the database of it's objects.

          Next, before the beginning of the next suite(which in our case happens to be network server suite), we go through the CleanDatabaseTestSetup.setUp code. As explained earlier, one of the things that CleanDatabaseTestSetup.setUp does is to call CleanDatabaseTestSetup.cleanDatabase. Which again goes through the drop table logic as explained above. It will attempt to drop the existing tables
          in batch but that runs into 1st lock timeout for "APP"."CLOBTBL". Next, it tries to drop the table individually, and it runs into 2nd lock timeout. Then the control goes back to CleanDatabaseTestSetup.removeObjects which calls drop table again after it drops the constraints. This time around, the batch exception runs into 3rd lock timeout exception. With my understanding of
          the code, we should have run into 4th lock timeout trying to drop the table individually but for some reason, I don't see that 4th lock time out with DRDA driver in derby.log. In any case, we once again exit
          CleanDatabaseTestSetup.cleanDatabase without successfully getting rid of all the tables and next step in CleanDatabaseTestSetup.setUp is the call to decorateSQL().decorateSQL fails when it tries to create tables but they already exist in the database becaue CleanDatabaseTestSetup.cleanDatabase was not able to get rid of them and that is why we get the exception Table/View 'BLOBTBL' already exists in Schema 'APP'.

          I am not sure why this locking issue is only intermittent but atleast we understand what is happening during the suite runs.

          Not sure if it is correct for CleanDatabaseTestSetup.cleanDatabase to run into exceptions while doing the cleanup but ignore those exceptions. I ran into a very old jira DERBY-2220(Uncommitted transactions executed throught XAResource will held locks after the application terminates (or crashes during the transaction).) where there was a brief conversation about how may be CleanDatabaseTestSetup.cleanDatabase should drop the database if it is not able to drop the objects inside the database. Here is part of Dan Debrunner's comment on that issue
          ******************
          I also think that you've found an issue in the clean database decorator and as
          such it would be good to fix it centrally there, rather than in a single test.
          for example, if the clean database decorator failed to cleanup the database,
          it could report the failure and then blow away the database.
          ******************

          We can do atleast one of the two things for this particular jira at this point (or may be even both)
          1)Have CleanDatabaseTestSetup.cleanDatabase report that it is having trouble deleting the objects and hence it will drop the database and create the objects mentioned in decorateSQL. This will get rid of intermittent failures in large data suite but it will not explain why do we intermittently run into lock timeouts
          2)Debug why we are running into intermittently lock timeouts with large data suite. I am running the tests with derby.locks.deadlockTrace=true to see if it will give us more information about the locks.

          Please let me know if there are any other options we could be pursuing.

          Show
          Mamta A. Satoor added a comment - I debugged more to see why we get 5 lock timeouts with embedded driver and then 3 locks with netwrok server before running into intermittent failure. My initial thought was that we will get 5 lock time outs both with embedded and network server before we give up trying to drop the tables. Here is the general flow chart from dropping tables point of view when the suite is using CleanDatabaseTestSetup. At the end of the last test fixture in the suite, CleanDatabaseTestSetup.tearDown gets called which does following(focusing on drop table related code, ignoring the code related to dropping aliases, views etc) CleanDatabaseTestSetup.tearDown-> CleanDatabaseTestSetup.cleanDatabase-> CleanDatabaseTestSetup.removeObjects-> JDBC.dropSchema-> .... // Tables rs = dmd.getTables((String) null, schema, (String) null, GET_TABLES_TABLE); dropUsingDMD(s, rs, schema, "TABLE_NAME", "TABLE"); .... // At this point there may be tables left due to // foreign key constraints leading to a dependency loop. // Drop any constraints that remain and then drop the tables. // If there are no tables then this should be a quick no-op. ..... // Tables (again) rs = dmd.getTables((String) null, schema, (String) null, GET_TABLES_TABLE); dropUsingDMD(s, rs, schema, "TABLE_NAME", "TABLE"); The code in dropUsingDMD is as follows First try to drop the objects in a batch rather than individually.If there is any BatchUpdateException while dropping the objects as a batch, drop the remaining objects inidividually in a for loop. If the for loop gets an exception but some of the objects before the exception got dropped successfully, then go back into the for loop and try to drop remaining objects again. This continues until either a)All objects are dropped and we can return b)no objects can be dropped because of exception After this CleanDatabaseTestSetup.tearDown is finished(with or without success), the control goes to the CleanDatabaseTestSetup.setUp for the next suite. Among other things, CleanDatabaseTestSetup.setUp does following 2 main steps 1)call CleanDatabaseTestSetup.cleanDatabase (drop any objects in the db) 2)call decorateSQL(create new objects in the db) Now, let's go through this code in our intermittent failure case. The last test from the embedded suite LobLimitsTest finishes with no errors and we go to CleanDatabaseTestSetup.tearDown(). This eventually leads to JDBC.dropUsingDMD as shown above. In JDBC.dropUsingDMD(), we first try to drop all the tables as a batch rather than individual drops. Here, we get the first lock timeout. Since we failed here, next we go in a loop to drop the tables individually. Here we manage to drop "APP"."CLOBTBL2" but get lock time out(2nd one) for trying to drop "APP"."CLOBTBL". But since the loop was able to drop at least one table, we go back into the loop to try to drop remaining tables. But that results into 3rd lock timeout for "APP"."CLOBTBL". After these tries, the control goes back to JDBC.dropSchema where we try to drop the tables yet again after dropping any constraints which may be preventing the table drops(in our case, I don't believe we have any constraints in the db). This 2nd attempt at dropping the table takes us back to JDBC.dropUsingDMD.Again we try to drop all the tables as batch but get the 4th lock timeout for "APP"."CLOBTBL". Next, we try to drop the tables inidividually in a loop, but the drop of "APP"."CLOBTBL" fails with 5th lock timeout. And since we were not able to drop any table in the loop, we exit out of the loop and eventually exit out of CleanDatabaseTestSetup.tearDown without reporting all this as a failure even though we were not able to clean the database of it's objects. Next, before the beginning of the next suite(which in our case happens to be network server suite), we go through the CleanDatabaseTestSetup.setUp code. As explained earlier, one of the things that CleanDatabaseTestSetup.setUp does is to call CleanDatabaseTestSetup.cleanDatabase. Which again goes through the drop table logic as explained above. It will attempt to drop the existing tables in batch but that runs into 1st lock timeout for "APP"."CLOBTBL". Next, it tries to drop the table individually, and it runs into 2nd lock timeout. Then the control goes back to CleanDatabaseTestSetup.removeObjects which calls drop table again after it drops the constraints. This time around, the batch exception runs into 3rd lock timeout exception. With my understanding of the code, we should have run into 4th lock timeout trying to drop the table individually but for some reason, I don't see that 4th lock time out with DRDA driver in derby.log. In any case, we once again exit CleanDatabaseTestSetup.cleanDatabase without successfully getting rid of all the tables and next step in CleanDatabaseTestSetup.setUp is the call to decorateSQL().decorateSQL fails when it tries to create tables but they already exist in the database becaue CleanDatabaseTestSetup.cleanDatabase was not able to get rid of them and that is why we get the exception Table/View 'BLOBTBL' already exists in Schema 'APP'. I am not sure why this locking issue is only intermittent but atleast we understand what is happening during the suite runs. Not sure if it is correct for CleanDatabaseTestSetup.cleanDatabase to run into exceptions while doing the cleanup but ignore those exceptions. I ran into a very old jira DERBY-2220 (Uncommitted transactions executed throught XAResource will held locks after the application terminates (or crashes during the transaction).) where there was a brief conversation about how may be CleanDatabaseTestSetup.cleanDatabase should drop the database if it is not able to drop the objects inside the database. Here is part of Dan Debrunner's comment on that issue ****************** I also think that you've found an issue in the clean database decorator and as such it would be good to fix it centrally there, rather than in a single test. for example, if the clean database decorator failed to cleanup the database, it could report the failure and then blow away the database. ****************** We can do atleast one of the two things for this particular jira at this point (or may be even both) 1)Have CleanDatabaseTestSetup.cleanDatabase report that it is having trouble deleting the objects and hence it will drop the database and create the objects mentioned in decorateSQL. This will get rid of intermittent failures in large data suite but it will not explain why do we intermittently run into lock timeouts 2)Debug why we are running into intermittently lock timeouts with large data suite. I am running the tests with derby.locks.deadlockTrace=true to see if it will give us more information about the locks. Please let me know if there are any other options we could be pursuing.
          Hide
          Mamta A. Satoor added a comment -

          I am attaching derby.log(named as derbyallFailWithPrintlnTrace.log) from today's failure with tracing on in the test code. The corresponding console output for it was attached earlier as CompleteRunallWithPrintlnTrace.out

          Show
          Mamta A. Satoor added a comment - I am attaching derby.log(named as derbyallFailWithPrintlnTrace.log) from today's failure with tracing on in the test code. The corresponding console output for it was attached earlier as CompleteRunallWithPrintlnTrace.out
          Hide
          Mamta A. Satoor added a comment -

          Just to be little more specific, I believe it is not the first test in the next suite that is failing, rather it is the setup that, the suite is supposed to do before firing any tests in the suite, is failing. So, (net)largedata.LobLimitsTest.test_01_Blob is not failing because nothing in that test fixture has started running. It is the setup (which attempts to drop the database objects and then create objects) is failing.

          One thing I am not sure about in case of intermittent failures, why do we get only three lock timeouts with rying to drop objects through network server rather than 5 with embedded. CleanDatabaseTestSetup.removeObjects() has a loop to try to drop objects 5 times. If there is no error on any of those attempts, then we simply quit from that loop, otherwise we throw the exception.

          I will put little more debugging code and fire the test again to see what is happening inside CleanDatabaseTestSetup.removeObjects() when we are going through different iterations etc. Will report more on what I find.

          Mike, I agree with you that I don't yet understand either why the problem is intermittent and why do we simply stop when we get table can't be dropped. Maybe we have coded the system such that if the setup for a suite fails, then don't bother running any fixture in that suite.

          Show
          Mamta A. Satoor added a comment - Just to be little more specific, I believe it is not the first test in the next suite that is failing, rather it is the setup that, the suite is supposed to do before firing any tests in the suite, is failing. So, (net)largedata.LobLimitsTest.test_01_Blob is not failing because nothing in that test fixture has started running. It is the setup (which attempts to drop the database objects and then create objects) is failing. One thing I am not sure about in case of intermittent failures, why do we get only three lock timeouts with rying to drop objects through network server rather than 5 with embedded. CleanDatabaseTestSetup.removeObjects() has a loop to try to drop objects 5 times. If there is no error on any of those attempts, then we simply quit from that loop, otherwise we throw the exception. I will put little more debugging code and fire the test again to see what is happening inside CleanDatabaseTestSetup.removeObjects() when we are going through different iterations etc. Will report more on what I find. Mike, I agree with you that I don't yet understand either why the problem is intermittent and why do we simply stop when we get table can't be dropped. Maybe we have coded the system such that if the setup for a suite fails, then don't bother running any fixture in that suite.
          Hide
          Mamta A. Satoor added a comment -

          large data test just finished with tracing on. It ran into the intermittent problem again(out of about 10 runs or so, I have gotten into intermittent failure twice). I am attaching the runall.out from this failed run with the tracing on. The attached file is named CompleteRunallWithPrintlnTrace.out

          Show
          Mamta A. Satoor added a comment - large data test just finished with tracing on. It ran into the intermittent problem again(out of about 10 runs or so, I have gotten into intermittent failure twice). I am attaching the runall.out from this failed run with the tracing on. The attached file is named CompleteRunallWithPrintlnTrace.out
          Hide
          Mike Matrigali added a comment -

          thanks mamta for the explanation. So for the original problem report from myrna we have:
          (emb)largedata.Derby5624Test.testDERBY_5624 used 518403 ms .
          (emb)largedata.LobLimitsTest.test_01_Blob used 2422 ms .
          (emb)largedata.LobLimitsTest.test_02_BlobNegative used 31 ms .
          (emb)largedata.LobLimitsTest.test_03_Clob1 used 2375 ms .
          (emb)largedata.LobLimitsTest.test_04_Clob2 used 3234 ms .
          (emb)largedata.LobLimitsTest.test_05_ClobNegative used 516 ms .
          (net)largedata.LobLimitsTest.test_01_Blob used 5360 ms .
          (net)largedata.LobLimitsTest.test_02_BlobNegative used 32 ms .
          (net)largedata.LobLimitsTest.test_03_Clob1 used 2078 ms .
          (net)largedata.LobLimitsTest.test_04_Clob2 used 2390 ms .
          (net)largedata.LobLimitsTest.test_05_ClobNegative used 938 ms .
          (emb)largedata.LobLimitsTest.test_01_Blob used 9188238 ms .
          (emb)largedata.LobLimitsTest.test_02_BlobNegative used 109 ms .
          (emb)largedata.LobLimitsTest.test_03_Clob1 used 8116714 ms .
          (emb)largedata.LobLimitsTest.test_04_Clob2 used 2164002 ms .
          (emb)largedata.LobLimitsTest.test_05_ClobNegative used 685745 ms E

          The failure is in a setup and has network server in its stack so does seem like next unlist test is failing and causing none of them to be run
          and to not report that they are failing.

          ok, i think your opinion is that it is actually the next test failing that is not listed (net) largedata.LoblimitsTest. I still don't understand why the problem is intermittent and why the first (net)largedata test run does not fail because the previous (emb) largedata.LobLimitsTest.test_05_ClobNegative didn't have an abort. I thought in this output a . meant success and and E or F meant failure in the listed test - i guess errors in the teardown/setup are not handled well.

          Show
          Mike Matrigali added a comment - thanks mamta for the explanation. So for the original problem report from myrna we have: (emb)largedata.Derby5624Test.testDERBY_5624 used 518403 ms . (emb)largedata.LobLimitsTest.test_01_Blob used 2422 ms . (emb)largedata.LobLimitsTest.test_02_BlobNegative used 31 ms . (emb)largedata.LobLimitsTest.test_03_Clob1 used 2375 ms . (emb)largedata.LobLimitsTest.test_04_Clob2 used 3234 ms . (emb)largedata.LobLimitsTest.test_05_ClobNegative used 516 ms . (net)largedata.LobLimitsTest.test_01_Blob used 5360 ms . (net)largedata.LobLimitsTest.test_02_BlobNegative used 32 ms . (net)largedata.LobLimitsTest.test_03_Clob1 used 2078 ms . (net)largedata.LobLimitsTest.test_04_Clob2 used 2390 ms . (net)largedata.LobLimitsTest.test_05_ClobNegative used 938 ms . (emb)largedata.LobLimitsTest.test_01_Blob used 9188238 ms . (emb)largedata.LobLimitsTest.test_02_BlobNegative used 109 ms . (emb)largedata.LobLimitsTest.test_03_Clob1 used 8116714 ms . (emb)largedata.LobLimitsTest.test_04_Clob2 used 2164002 ms . (emb)largedata.LobLimitsTest.test_05_ClobNegative used 685745 ms E The failure is in a setup and has network server in its stack so does seem like next unlist test is failing and causing none of them to be run and to not report that they are failing. ok, i think your opinion is that it is actually the next test failing that is not listed (net) largedata.LoblimitsTest. I still don't understand why the problem is intermittent and why the first (net)largedata test run does not fail because the previous (emb) largedata.LobLimitsTest.test_05_ClobNegative didn't have an abort. I thought in this output a . meant success and and E or F meant failure in the listed test - i guess errors in the teardown/setup are not handled well.
          Hide
          Mamta A. Satoor added a comment -

          I am attaching what a successful run of large data run will look like when the output of the suite has been directed to a file as shown below
          time java -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1

          Show
          Mamta A. Satoor added a comment - I am attaching what a successful run of large data run will look like when the output of the suite has been directed to a file as shown below time java -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1
          Hide
          Mamta A. Satoor added a comment -

          I have attached the code changes to have some debugging into DERBY5638_patch2_diff.txt. I have also attached runallWithPrintlnTrace.out which is the output of the following large data run with debugging changes
          $ time java -Dderby.tests.debug=true -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1

          The large data suite is not finished yet and hence runallWithPrintlnTrace.out is not the complete run but it has debugging info to get an idea of when setup and teardown get run.

          Show
          Mamta A. Satoor added a comment - I have attached the code changes to have some debugging into DERBY5638_patch2_diff.txt. I have also attached runallWithPrintlnTrace.out which is the output of the following large data run with debugging changes $ time java -Dderby.tests.debug=true -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1 The large data suite is not finished yet and hence runallWithPrintlnTrace.out is not the complete run but it has debugging info to get an idea of when setup and teardown get run.
          Hide
          Mamta A. Satoor added a comment -

          Mike commented "And to me it looks like the intermittent error happens while setting up to run test_05_ClobNegative. But your fix seems to be to change logic at the end of test_05_ClobNegative. "

          My understanding of CleanDatabaseTestSetup is that it runs the steup at the beginning of the suite run and it runs the teardown at the end of the suite after the last test has run. And these same steps repeat for every suite. I have put couple printlns in CleanDatabaseTestSetup to see this sequence while running large data suite. To see these printlns, we need to pass derby.tests.trace=true as shown below
          $ time java -Dderby.tests.debug=true -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1
          The use of Dderby.tests.trace=true puts all the println to the console(which I have redirected to runall.out in my case)

          The tests have not finished running(I will attach runall.out with what I have so far. It is very verbose because all the existing printlns will now get printed because of trace flag. You can search for "DEBUG: cleanDatabase" in that file to see when setup and teardown gets called). I can see that debug println for setup happens before the suite is run. Then after the suite is finished, teardown is run before the new suite is started. And so on and so forth.
          eg of such a sequence (largedata.Derby5624Test.testDERBY_5624 is the first suite and it has just one fixture. largedata.LobLimitsTest.test_01_Blob is start of a new suite.)

          DEBUG: cleanDatabase start in setup
          DEBUG: cleanDatabase end in setup
          .
          (emb)largedata.Derby5624Test.testDERBY_5624 used 299858 ms DEBUG: cleanDatabase start in teardown
          DEBUG: cleanDatabase end in teardown
          DEBUG: cleanDatabase start in setup
          DEBUG: cleanDatabase end in setup
          DEBUG: BIGGEST_LOB_SZ=1048576 BIG_LOB_SZ=102400
          .
          (emb)largedata.LobLimitsTest.test_01_Blob DEBUG: ========================================

          An interesting eg of the setup and teardown can be seen when the last test in embedded suite has run and when the next suite is getting run in network server(largedata.LobLimitsTest.test_05_ClobNegative is the last test in the previous suite. largedata.LobLimitsTest.test_01_Blob is the first fixture for next suite which is a network server suite)
          (emb)largedata.LobLimitsTest.test_05_ClobNegative DEBUG: at the beginning of test_05_ClobNegative
          DEBUG: ========================================
          DEBUG: START ClobTest #7insert Clob of size = 102400
          DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@38203820
          DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@38203820
          DEBUG: ========================================
          DEBUG: START ClobTest #7insert Clob of size = 102400
          DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@46034603
          DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@46034603
          DEBUG: Got FileWriter for extinout/charLobLimits.txt java.io.FileWriter@34243424
          DEBUG: writer java.io.FileWriter@34243424 for file extinout/charLobLimits.txt closed
          DEBUG: ========================================
          DEBUG: START ClobTest #9.1 insert Clob of size = 102401
          DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@5e015e01
          DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@5e015e01
          DEBUG: ========================================
          DEBUG: START ClobTest #9.2 - SELECT CLOB of size = 102400
          DEBUG: Matched rows selected with clob of size(102400) =0
          DEBUG: Select Clob (102400) rows= 0 = 7
          DEBUG: ========================================
          DEBUG: ========================================
          DEBUG: START ClobTest #10 insert Clob of size = 102401
          DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@35f135f1
          DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@35f135f1
          DEBUG: ========================================
          DEBUG: START ClobTest #11 insert Clob of size = 102401
          DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@77f277f2
          DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@77f277f2
          DEBUG: Rows deleted =2
          DEBUG: ========================================
          DEBUG: START ClobTest #12.1 -insertClob of size = 102400
          DEBUG: ========================================
          DEBUG: START ClobTest #12.2 - SELECT CLOB of size = 102400
          DEBUG: Select Clob (102400) rows= 0 = 1
          DEBUG: Matched rows selected with clob of size(102400) =0
          DEBUG: ========================================
          DEBUG: Rows deleted =2
          DEBUG: ========================================
          DEBUG: START ClobTest #13 (setClob with 4Gb clobinsertClob of size = 4294967296
          DEBUG: at the end of test_05_ClobNegative
          used 506 ms DEBUG: cleanDatabase start in teardown
          DEBUG: cleanDatabase end in teardown
          DEBUG: Starting network server:
          DEBUG: cleanDatabase start in setup
          DEBUG: cleanDatabase end in setup
          DEBUG: BIGGEST_LOB_SZ=1048576 BIG_LOB_SZ=102400
          .
          (net)largedata.LobLimitsTest.test_01_Blob DEBUG: ========================================

          So based on the above information, I believe that the intermittent error happens after the last test test_05_ClobNegative in the embedded suite has finished running and while in the teardown code, it keeps running into lock time out exceptions(the drop objects is tried 5 times and we see 5 lock timeouts). Then the network server is started, it goes through the setup method. The setup method in CleanDatabaseTestSetup looks as follows
          protected void setUp() throws Exception {
          if (jdbcClient != null )

          { // We have network client (useNetworkClient) on a given host and port. TestConfiguration current = TestConfiguration.getCurrent(); TestConfiguration modified = new TestConfiguration(current, jdbcClient, hostName, portNo); TestConfiguration.setCurrent(modified); }

          Connection conn = getConnection();
          conn.setAutoCommit(false);

          // compress as well to allow the fixtures wrapped in
          // this decorator to start with a clean database.
          println("cleanDatabase start in setup");
          CleanDatabaseTestSetup.cleanDatabase(conn, true);
          println("cleanDatabase end in setup");

          Statement s = conn.createStatement();
          decorateSQL(s);

          s.close();
          conn.commit();
          conn.close();
          }

          setup() first does the drop of objects again and then goes through the decorateSQL to create test obects again. This is where in failed intermittent failures, I believe we start seeing more lock time outs for drop table but with network server driver(because the new suite is network server suite). And subsequent try to create the table fails because we were never able to drop the tables successfully because of lock timeouts.

          Sorry for this being such a long comment. Please ask me questions if this is not clear. I will post svn diff for simple printlns I put for tracking when setup and teardowns are getting called. I will also put the runall.out for what has finished so far with these extra printlns.

          Show
          Mamta A. Satoor added a comment - Mike commented "And to me it looks like the intermittent error happens while setting up to run test_05_ClobNegative. But your fix seems to be to change logic at the end of test_05_ClobNegative. " My understanding of CleanDatabaseTestSetup is that it runs the steup at the beginning of the suite run and it runs the teardown at the end of the suite after the last test has run. And these same steps repeat for every suite. I have put couple printlns in CleanDatabaseTestSetup to see this sequence while running large data suite. To see these printlns, we need to pass derby.tests.trace=true as shown below $ time java -Dderby.tests.debug=true -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1 The use of Dderby.tests.trace=true puts all the println to the console(which I have redirected to runall.out in my case) The tests have not finished running(I will attach runall.out with what I have so far. It is very verbose because all the existing printlns will now get printed because of trace flag. You can search for "DEBUG: cleanDatabase" in that file to see when setup and teardown gets called). I can see that debug println for setup happens before the suite is run. Then after the suite is finished, teardown is run before the new suite is started. And so on and so forth. eg of such a sequence (largedata.Derby5624Test.testDERBY_5624 is the first suite and it has just one fixture. largedata.LobLimitsTest.test_01_Blob is start of a new suite.) DEBUG: cleanDatabase start in setup DEBUG: cleanDatabase end in setup . (emb)largedata.Derby5624Test.testDERBY_5624 used 299858 ms DEBUG: cleanDatabase start in teardown DEBUG: cleanDatabase end in teardown DEBUG: cleanDatabase start in setup DEBUG: cleanDatabase end in setup DEBUG: BIGGEST_LOB_SZ=1048576 BIG_LOB_SZ=102400 . (emb)largedata.LobLimitsTest.test_01_Blob DEBUG: ======================================== An interesting eg of the setup and teardown can be seen when the last test in embedded suite has run and when the next suite is getting run in network server(largedata.LobLimitsTest.test_05_ClobNegative is the last test in the previous suite. largedata.LobLimitsTest.test_01_Blob is the first fixture for next suite which is a network server suite) (emb)largedata.LobLimitsTest.test_05_ClobNegative DEBUG: at the beginning of test_05_ClobNegative DEBUG: ======================================== DEBUG: START ClobTest #7insert Clob of size = 102400 DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@38203820 DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@38203820 DEBUG: ======================================== DEBUG: START ClobTest #7insert Clob of size = 102400 DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@46034603 DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@46034603 DEBUG: Got FileWriter for extinout/charLobLimits.txt java.io.FileWriter@34243424 DEBUG: writer java.io.FileWriter@34243424 for file extinout/charLobLimits.txt closed DEBUG: ======================================== DEBUG: START ClobTest #9.1 insert Clob of size = 102401 DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@5e015e01 DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@5e015e01 DEBUG: ======================================== DEBUG: START ClobTest #9.2 - SELECT CLOB of size = 102400 DEBUG: Matched rows selected with clob of size(102400) =0 DEBUG: Select Clob (102400) rows= 0 = 7 DEBUG: ======================================== DEBUG: ======================================== DEBUG: START ClobTest #10 insert Clob of size = 102401 DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@35f135f1 DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@35f135f1 DEBUG: ======================================== DEBUG: START ClobTest #11 insert Clob of size = 102401 DEBUG: Got reader for file extinout/charLobLimits.txt java.io.FileReader@77f277f2 DEBUG: Closed reader for file extinout/charLobLimits.txt java.io.FileReader@77f277f2 DEBUG: Rows deleted =2 DEBUG: ======================================== DEBUG: START ClobTest #12.1 -insertClob of size = 102400 DEBUG: ======================================== DEBUG: START ClobTest #12.2 - SELECT CLOB of size = 102400 DEBUG: Select Clob (102400) rows= 0 = 1 DEBUG: Matched rows selected with clob of size(102400) =0 DEBUG: ======================================== DEBUG: Rows deleted =2 DEBUG: ======================================== DEBUG: START ClobTest #13 (setClob with 4Gb clobinsertClob of size = 4294967296 DEBUG: at the end of test_05_ClobNegative used 506 ms DEBUG: cleanDatabase start in teardown DEBUG: cleanDatabase end in teardown DEBUG: Starting network server: DEBUG: cleanDatabase start in setup DEBUG: cleanDatabase end in setup DEBUG: BIGGEST_LOB_SZ=1048576 BIG_LOB_SZ=102400 . (net)largedata.LobLimitsTest.test_01_Blob DEBUG: ======================================== So based on the above information, I believe that the intermittent error happens after the last test test_05_ClobNegative in the embedded suite has finished running and while in the teardown code, it keeps running into lock time out exceptions(the drop objects is tried 5 times and we see 5 lock timeouts). Then the network server is started, it goes through the setup method. The setup method in CleanDatabaseTestSetup looks as follows protected void setUp() throws Exception { if (jdbcClient != null ) { // We have network client (useNetworkClient) on a given host and port. TestConfiguration current = TestConfiguration.getCurrent(); TestConfiguration modified = new TestConfiguration(current, jdbcClient, hostName, portNo); TestConfiguration.setCurrent(modified); } Connection conn = getConnection(); conn.setAutoCommit(false); // compress as well to allow the fixtures wrapped in // this decorator to start with a clean database. println("cleanDatabase start in setup"); CleanDatabaseTestSetup.cleanDatabase(conn, true); println("cleanDatabase end in setup"); Statement s = conn.createStatement(); decorateSQL(s); s.close(); conn.commit(); conn.close(); } setup() first does the drop of objects again and then goes through the decorateSQL to create test obects again. This is where in failed intermittent failures, I believe we start seeing more lock time outs for drop table but with network server driver(because the new suite is network server suite). And subsequent try to create the table fails because we were never able to drop the tables successfully because of lock timeouts. Sorry for this being such a long comment. Please ask me questions if this is not clear. I will post svn diff for simple printlns I put for tracking when setup and teardowns are getting called. I will also put the runall.out for what has finished so far with these extra printlns.
          Hide
          Mamta A. Satoor added a comment -

          Mike commented "could you post the derby.log containing the lock timeouts that remain. I assume that we only know all the lock timeouts on the last suite, as we lose the derby.log from previous one. "

          I just attached the log as derbyWithRollbackInTest_05.log

          Show
          Mamta A. Satoor added a comment - Mike commented "could you post the derby.log containing the lock timeouts that remain. I assume that we only know all the lock timeouts on the last suite, as we lose the derby.log from previous one. " I just attached the log as derbyWithRollbackInTest_05.log
          Hide
          Mike Matrigali added a comment -

          The change is probably a good one, but I am not following how it will fix the problem being reported. I am assuming the lexical order of the
          tests being run is:
          public void test_01_Blob() throws Exception {
          public void test_02_BlobNegative() throws SQLException {
          public void test_03_Clob1() throws Exception {
          public void test_04_Clob2() throws Exception {
          public void test_05_ClobNegative() throws Exception {

          And to me it looks like the intermittent error happens while setting up to run test_05_ClobNegative. But your fix seems to be to change
          logic at the end of test_05_ClobNegative.

          Have you looked at each of the tests to determine if each should have a commit/abort in all exit paths of the test?

          Show
          Mike Matrigali added a comment - The change is probably a good one, but I am not following how it will fix the problem being reported. I am assuming the lexical order of the tests being run is: public void test_01_Blob() throws Exception { public void test_02_BlobNegative() throws SQLException { public void test_03_Clob1() throws Exception { public void test_04_Clob2() throws Exception { public void test_05_ClobNegative() throws Exception { And to me it looks like the intermittent error happens while setting up to run test_05_ClobNegative. But your fix seems to be to change logic at the end of test_05_ClobNegative. Have you looked at each of the tests to determine if each should have a commit/abort in all exit paths of the test?
          Hide
          Mike Matrigali added a comment -

          could you post the derby.log containing the lock timeouts that remain. I assume that we only know all the lock timeouts on the last suite, as we lose the derby.log from previous one.

          Does it make sense to anyone that the test is not reporting the lock timeouts as errors on the run. I am pretty sure that the database cleanup
          code reports junit errors when it retries but can't remove files. Was expecting the same on lock timeouts, unless it eventually succeeds on
          the retries.

          Show
          Mike Matrigali added a comment - could you post the derby.log containing the lock timeouts that remain. I assume that we only know all the lock timeouts on the last suite, as we lose the derby.log from previous one. Does it make sense to anyone that the test is not reporting the lock timeouts as errors on the run. I am pretty sure that the database cleanup code reports junit errors when it retries but can't remove files. Was expecting the same on lock timeouts, unless it eventually succeeds on the retries.
          Hide
          Mamta A. Satoor added a comment -

          With the addition of rollback in LobLimitsTest.test_05_ClobNegative fixture, I ran the large data suite twice and it passed both those times. There were still lock time out exceptions in derby.log but only 3 in both those runs compared to 6 that I saw in the earlier successful run(without the rollback change). I will go ahead and commit the rollback change into trunk.

          Show
          Mamta A. Satoor added a comment - With the addition of rollback in LobLimitsTest.test_05_ClobNegative fixture, I ran the large data suite twice and it passed both those times. There were still lock time out exceptions in derby.log but only 3 in both those runs compared to 6 that I saw in the earlier successful run(without the rollback change). I will go ahead and commit the rollback change into trunk.
          Hide
          Mamta A. Satoor added a comment -

          The derby.log for the passed and failed runs will show that they both have lock time outs while trying to drop the tables to get the database ready in a clean state again. The failed version has more lock time outs than the passing version.

          It appears that CleanDatabaseTestSetup tries 5 times to drop database objects. For both passed and failed versions, all those 5 attempts fail with lock time outs. Then for some reason, in both failed and passed version, it appears that we try to drop the objects in network server mode(this is what I see in derby.log for both runs. I haven't looked for where it actually happens in the junit code). Both passing and failing derby.log show lock time out for drop table through network server mode. But after this failure, the passing derby.log continues to finish the large data suite with no problem. But with fail version, we get 2 more lock timeouts for trying to drop the table in network server mode. after those lock time outs, we get table already exists error in case of failing derby.log

          Some background info on largedata._Suite
          largedata._Suite runs following suits
          Derby5624Test(only f windows platform)
          LobLimitsLiteTest
          LobLimitsTest
          LobLimitsClientTest

          The intermittent problem we are seeing happens while running LobLimitsTest suite. LobLimitsTest has 5 test fixtures which get run in specific order(order is ensured through TestConfiguration.orderedSuite). The intermittent problem is in the last of those 5 fixtures, namely test_05_ClobNegative. As the name indicates, this test does a bunch of negative tests but it does not do a rollback or commit at the end. test_02_BlobNegative() is the 2nd of the 5 test fixtures run by LobLimitsTest and it does negative testing too but at the end, in the catch block for the expected exception, it does a commit.

          I think we may be seeing lock time outs because the transaction started by test_05_ClobNegative was never committed/rolled back and hence the locks were never released. I am going to make just one line change in my codeline to LobLimitsTest and add rollback() to the end of test_05_ClobNegative. I will run largedata test on my machine with that change to see how it goes. I will attach that one line change patch for reference to this jira.

          Show
          Mamta A. Satoor added a comment - The derby.log for the passed and failed runs will show that they both have lock time outs while trying to drop the tables to get the database ready in a clean state again. The failed version has more lock time outs than the passing version. It appears that CleanDatabaseTestSetup tries 5 times to drop database objects. For both passed and failed versions, all those 5 attempts fail with lock time outs. Then for some reason, in both failed and passed version, it appears that we try to drop the objects in network server mode(this is what I see in derby.log for both runs. I haven't looked for where it actually happens in the junit code). Both passing and failing derby.log show lock time out for drop table through network server mode. But after this failure, the passing derby.log continues to finish the large data suite with no problem. But with fail version, we get 2 more lock timeouts for trying to drop the table in network server mode. after those lock time outs, we get table already exists error in case of failing derby.log Some background info on largedata._Suite largedata._Suite runs following suits Derby5624Test(only f windows platform) LobLimitsLiteTest LobLimitsTest LobLimitsClientTest The intermittent problem we are seeing happens while running LobLimitsTest suite. LobLimitsTest has 5 test fixtures which get run in specific order(order is ensured through TestConfiguration.orderedSuite). The intermittent problem is in the last of those 5 fixtures, namely test_05_ClobNegative. As the name indicates, this test does a bunch of negative tests but it does not do a rollback or commit at the end. test_02_BlobNegative() is the 2nd of the 5 test fixtures run by LobLimitsTest and it does negative testing too but at the end, in the catch block for the expected exception, it does a commit. I think we may be seeing lock time outs because the transaction started by test_05_ClobNegative was never committed/rolled back and hence the locks were never released. I am going to make just one line change in my codeline to LobLimitsTest and add rollback() to the end of test_05_ClobNegative. I will run largedata test on my machine with that change to see how it goes. I will attach that one line change patch for reference to this jira.
          Hide
          Mamta A. Satoor added a comment -

          Mike asked "Have you changed the test at all in these runs to get more information? If so can you check those changes in? "
          No, I haven't changed anything in the test to get more information. I just fired the tests using
          $ time java -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1

          Show
          Mamta A. Satoor added a comment - Mike asked "Have you changed the test at all in these runs to get more information? If so can you check those changes in? " No, I haven't changed anything in the test to get more information. I just fired the tests using $ time java -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.largedata._Suite > runall.out 2>&1
          Hide
          Mike Matrigali added a comment -

          it looks like in the failed run you are getting lock timeouts during:
          CleanDatabaseTestSetup.tearDown

          This is causing the drop's to fail, but I don't know why these errors are not getting reported in the suite error log - this seems like a bug either
          in the tests or the test utilities. Should make sure this test
          is using the same mechanism being used in suites.all to retry on failed drops, i think that stuff mostly retries if deletes fail I don't know if it
          does anything on timeouts.

          I would also suggest updating the test to set the higher lock diagnostics so that we can see more of what is going on in the timeout. I might
          expect a very short timeout to conflict with the background tasks, but not normal timeout. But maybe the data set is so huge with the blobs
          that there is so much work to be done that it does not allow the drop to happen. Should figure out what timeout is being used for the drop. An interesting enhancement to the lock timeout message would be to add the default timeout being used.

          If the test sets timeout anywhere should make sure it is getting set back before the cleanup.

          Show
          Mike Matrigali added a comment - it looks like in the failed run you are getting lock timeouts during: CleanDatabaseTestSetup.tearDown This is causing the drop's to fail, but I don't know why these errors are not getting reported in the suite error log - this seems like a bug either in the tests or the test utilities. Should make sure this test is using the same mechanism being used in suites.all to retry on failed drops, i think that stuff mostly retries if deletes fail I don't know if it does anything on timeouts. I would also suggest updating the test to set the higher lock diagnostics so that we can see more of what is going on in the timeout. I might expect a very short timeout to conflict with the background tasks, but not normal timeout. But maybe the data set is so huge with the blobs that there is so much work to be done that it does not allow the drop to happen. Should figure out what timeout is being used for the drop. An interesting enhancement to the lock timeout message would be to add the default timeout being used. If the test sets timeout anywhere should make sure it is getting set back before the cleanup.
          Hide
          Mike Matrigali added a comment -

          Have you changed the test at all in these runs to get more information? If so can you check those changes in?

          Show
          Mike Matrigali added a comment - Have you changed the test at all in these runs to get more information? If so can you check those changes in?
          Hide
          Mamta A. Satoor added a comment -

          I have been running large data suite on my Windows 7 laptop with IBM jdk 1.6 on trunk to see if I can reproduce the problem. The test failed for me one out of 5 runs so far. I am attaching derby.log from the failed time and from one of the successful runs. The failed derby.log is attached as derbyFailed and passed derby.log is attached as derbyPass.log. I will look through the 2 to see if anything jumps out in terms of what is the reason behind intermittent failures.

          Show
          Mamta A. Satoor added a comment - I have been running large data suite on my Windows 7 laptop with IBM jdk 1.6 on trunk to see if I can reproduce the problem. The test failed for me one out of 5 runs so far. I am attaching derby.log from the failed time and from one of the successful runs. The failed derby.log is attached as derbyFailed and passed derby.log is attached as derbyPass.log. I will look through the 2 to see if anything jumps out in terms of what is the reason behind intermittent failures.
          Hide
          Myrna van Lunteren added a comment -

          The test passed again on the linux/vmware machine where I saw this failure originally on 2/27, 2/28, 2/29, 3/1, 3/2 but failed again on 3/3 (and we got a disk full on 3/4)

          Show
          Myrna van Lunteren added a comment - The test passed again on the linux/vmware machine where I saw this failure originally on 2/27, 2/28, 2/29, 3/1, 3/2 but failed again on 3/3 (and we got a disk full on 3/4)
          Hide
          Myrna van Lunteren added a comment -

          Some additional info.

          I first started running this _Suite every night that there were changes (plus every Friday regardless of changes) with trunk and ibm 1.6 in September of 2011.
          It ran into a failure - ERROR 25502 - (An SQL data change is not permitted for a read-only connection, user, or database) a disk space issue on December 7, 2011, and after that (extending to Dec 8, 2011) the disk ran full. I cleaned that up, and tests ran fine again until 2/21/2012, when the tests failed because of an issue with the Derby5624Test. After that, I first saw this error (Table/View BLOBTBL already exists) on 2/22/2012. It seemed at the time this was possibly related, but that does not make sense - the only change was adding of a new test, and this test was afterwards getting skipped on linux.
          To confirm, I have run the suite on an exact matching machine with 10.8 (and ibm1.6) and there have been no failures.
          I also ran the suite on a much slower 1 CPU machine with windows XP, and trunk, and there, I have seen the test fail 4 out of 5 times (before old test results have clogged up my disk).

          Show
          Myrna van Lunteren added a comment - Some additional info. I first started running this _Suite every night that there were changes (plus every Friday regardless of changes) with trunk and ibm 1.6 in September of 2011. It ran into a failure - ERROR 25502 - (An SQL data change is not permitted for a read-only connection, user, or database) a disk space issue on December 7, 2011, and after that (extending to Dec 8, 2011) the disk ran full. I cleaned that up, and tests ran fine again until 2/21/2012, when the tests failed because of an issue with the Derby5624Test. After that, I first saw this error (Table/View BLOBTBL already exists) on 2/22/2012. It seemed at the time this was possibly related, but that does not make sense - the only change was adding of a new test, and this test was afterwards getting skipped on linux. To confirm, I have run the suite on an exact matching machine with 10.8 (and ibm1.6) and there have been no failures. I also ran the suite on a much slower 1 CPU machine with windows XP, and trunk, and there, I have seen the test fail 4 out of 5 times (before old test results have clogged up my disk).

            People

            • Assignee:
              Mamta A. Satoor
              Reporter:
              Myrna van Lunteren
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development