Derby
  1. Derby
  2. DERBY-803

derbynet/DerbyNetAutoStart.java test fails intermittently with org.apache.derby.iapi.services.context.ShutdownException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.1.3.2, 10.2.1.6, 10.3.1.4
    • Fix Version/s: 10.2.1.6
    • Component/s: Network Server
    • Labels:
      None
    • Bug behavior facts:
      Regression Test Failure

      Description

      DerbyNetAutoStart fails intermittently with the following diff:

      This issue is likely related to DERBY-1020

                      • Diff file derbyall/derbynetmats/DerbyNet/derbynetmats/DerbyNetAutoStart.diff
          • Start: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-01-05 23:39:40 ***
            1a2,3
            > org.apache.derby.iapi.services.context.ShutdownException:
            > at org.apache.derby.impl.drda.Session.close(Unknown Source)agentThread[DRDAConnThread_3,5,derby.daemons]
            Test Failed.
          • End: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-01-05 23:41:10 ***
      1. derby-803.diff
        0.7 kB
      2. derby-803.stat
        0.2 kB
      3. suggestion-803.diff
        3 kB
        Fernanda Pizzorno
      4. suggestion-803.stat
        0.3 kB
        Fernanda Pizzorno

        Issue Links

          Activity

          Hide
          Andrew McIntyre added a comment -

          Since this was a minor change to the test that fixed the problem, and this test is a bit of an annoyance for 10.1 testing, I merged Fernanda's test change to 10.1 with revision 462705.

          Show
          Andrew McIntyre added a comment - Since this was a minor change to the test that fixed the problem, and this test is a bit of an annoyance for 10.1 testing, I merged Fernanda's test change to 10.1 with revision 462705.
          Hide
          Sunitha Kambhampati added a comment -

          This test fails on 10.1.3.2 also. Does it make sense for this fix to be backported to 10.1 codeline ?

          Thanks.

          Show
          Sunitha Kambhampati added a comment - This test fails on 10.1.3.2 also. Does it make sense for this fix to be backported to 10.1 codeline ? Thanks.
          Hide
          Knut Anders Hatlen added a comment -

          Merged to 10.2 in Rick's mega-merge.

          Show
          Knut Anders Hatlen added a comment - Merged to 10.2 in Rick's mega-merge.
          Hide
          Knut Anders Hatlen added a comment -

          Thank you, Fernanda! Although your patch only works around the problem, I think the fix is OK. The general problem is logged as DERBY-1020, and until it has been fixed, I think it is good to have a workaround which makes this test run cleanly. I added a comment about why the sleep was needed and committed revision 442490.

          Leaving the issue open until it has been merged to 10.2.

          Show
          Knut Anders Hatlen added a comment - Thank you, Fernanda! Although your patch only works around the problem, I think the fix is OK. The general problem is logged as DERBY-1020 , and until it has been fixed, I think it is good to have a workaround which makes this test run cleanly. I added a comment about why the sleep was needed and committed revision 442490. Leaving the issue open until it has been merged to 10.2.
          Hide
          Fernanda Pizzorno added a comment -

          The attached patch (derby-803.diff) changes the test so that the exception will not happen. I have successfully run the DerbyNetAutoStart 100 times with this patch.

          Show
          Fernanda Pizzorno added a comment - The attached patch (derby-803.diff) changes the test so that the exception will not happen. I have successfully run the DerbyNetAutoStart 100 times with this patch.
          Hide
          Mike Matrigali added a comment -

          I am seeing the following similar to what Suresh reported, against an ibm1.4.2 jvm, against the trunk

          Generating report for RunSuite derbynetmats null null DerbyNet true
          ------------------ Java Information ------------------
          Java Version: 1.4.2
          Java Vendor: IBM Corporation
          Java home: c:\cloudtst\jartest\ibm142\jre
          Java classpath: c:/cloudtst/jartest/classes/derby.jar;c:/cloudtst/jartest/classes/derbytools.jar;c:/cloudtst/jartest/classes/derbynet.jar;c:/cloudtst/jartest/classes/derbyclient.jar;c:/cloudtst/jartest/classes/functionTests.jar;c:/cloudtst/jartest/classes/derbyTesting.jar;c:/cloudtst/jartest/tools/java/jndi/fscontext.jar;c:/cloudtst/jartest/drda/jcc/2.6_latest/db2jcc.jar;c:/cloudtst/jartest/drda/jcc/2.6_latest/db2jcc_license_c.jar;c:/cloudtst/jartest/tools/java/junit.jar;c:/cloudtst/jartest/classes/derbyLocale_zh_TW.jar;c:/cloudtst/jartest/classes/derbyLocale_zh_CN.jar;c:/cloudtst/jartest/classes/derbyLocale_pt_BR.jar;c:/cloudtst/jartest/classes/derbyLocale_ko_KR.jar;c:/cloudtst/jartest/classes/derbyLocale_ja_JP.jar;c:/cloudtst/jartest/classes/derbyLocale_it.jar;c:/cloudtst/jartest/classes/derbyLocale_fr.jar;c:/cloudtst/jartest/classes/derbyLocale_es.jar;c:/cloudtst/jartest/classes/derbyLocale_de_DE.jar;
          OS name: Windows XP
          OS architecture: x86
          OS version: 5.1
          Java user name: cloudtest
          Java user home: C:\Documents and Settings\cloudtest
          Java user dir: C:\cloudtst\jartest\JarResults.2006-09-06\ibm142_derbynetmats_jcc_2.6_latest
          java.specification.name: Java Platform API Specification
          java.specification.version: 1.4
          --------- Derby Information --------
          JRE - JDBC: J2SE 1.4.2 - JDBC 3.0
          [C:\cloudtst\jartest\classes\derby.jar] 10.3.0.0 alpha - (440936)
          [C:\cloudtst\jartest\classes\derbytools.jar] 10.3.0.0 alpha - (440936)
          [C:\cloudtst\jartest\classes\derbynet.jar] 10.3.0.0 alpha - (440936)
          [C:\cloudtst\jartest\classes\derbyClient.jar] 10.3.0.0 alpha - (440936)
          [C:\cloudtst\jartest\drda\jcc\2.6_latest\db2jcc.jar] 2.6 - (87)
          [C:\cloudtst\jartest\drda\jcc\2.6_latest\db2jcc_license_c.jar] 2.6 - (87)
          ------------------------------------------------------

              • Start: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-09-07 11:22:14 ***
                1a2,4
                > org.apache.derby.iapi.services.context.ShutdownException:
                > agentError in Thread[DRDAConnThread_5,5,derby.daemons]
                > org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = database1; diagnostic msg = Java exception: ': org.apache.derby.iapi.services.context.ShutdownException'.
                Test Failed.
              • End: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-09-07 11:23:11 ***
          Show
          Mike Matrigali added a comment - I am seeing the following similar to what Suresh reported, against an ibm1.4.2 jvm, against the trunk Generating report for RunSuite derbynetmats null null DerbyNet true ------------------ Java Information ------------------ Java Version: 1.4.2 Java Vendor: IBM Corporation Java home: c:\cloudtst\jartest\ibm142\jre Java classpath: c:/cloudtst/jartest/classes/derby.jar;c:/cloudtst/jartest/classes/derbytools.jar;c:/cloudtst/jartest/classes/derbynet.jar;c:/cloudtst/jartest/classes/derbyclient.jar;c:/cloudtst/jartest/classes/functionTests.jar;c:/cloudtst/jartest/classes/derbyTesting.jar;c:/cloudtst/jartest/tools/java/jndi/fscontext.jar;c:/cloudtst/jartest/drda/jcc/2.6_latest/db2jcc.jar;c:/cloudtst/jartest/drda/jcc/2.6_latest/db2jcc_license_c.jar;c:/cloudtst/jartest/tools/java/junit.jar;c:/cloudtst/jartest/classes/derbyLocale_zh_TW.jar;c:/cloudtst/jartest/classes/derbyLocale_zh_CN.jar;c:/cloudtst/jartest/classes/derbyLocale_pt_BR.jar;c:/cloudtst/jartest/classes/derbyLocale_ko_KR.jar;c:/cloudtst/jartest/classes/derbyLocale_ja_JP.jar;c:/cloudtst/jartest/classes/derbyLocale_it.jar;c:/cloudtst/jartest/classes/derbyLocale_fr.jar;c:/cloudtst/jartest/classes/derbyLocale_es.jar;c:/cloudtst/jartest/classes/derbyLocale_de_DE.jar; OS name: Windows XP OS architecture: x86 OS version: 5.1 Java user name: cloudtest Java user home: C:\Documents and Settings\cloudtest Java user dir: C:\cloudtst\jartest\JarResults.2006-09-06\ibm142_derbynetmats_jcc_2.6_latest java.specification.name: Java Platform API Specification java.specification.version: 1.4 --------- Derby Information -------- JRE - JDBC: J2SE 1.4.2 - JDBC 3.0 [C:\cloudtst\jartest\classes\derby.jar] 10.3.0.0 alpha - (440936) [C:\cloudtst\jartest\classes\derbytools.jar] 10.3.0.0 alpha - (440936) [C:\cloudtst\jartest\classes\derbynet.jar] 10.3.0.0 alpha - (440936) [C:\cloudtst\jartest\classes\derbyClient.jar] 10.3.0.0 alpha - (440936) [C:\cloudtst\jartest\drda\jcc\2.6_latest\db2jcc.jar] 2.6 - (87) [C:\cloudtst\jartest\drda\jcc\2.6_latest\db2jcc_license_c.jar] 2.6 - (87) ------------------------------------------------------ Start: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-09-07 11:22:14 *** 1a2,4 > org.apache.derby.iapi.services.context.ShutdownException: > agentError in Thread [DRDAConnThread_5,5,derby.daemons] > org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = database1; diagnostic msg = Java exception: ': org.apache.derby.iapi.services.context.ShutdownException'. Test Failed. End: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-09-07 11:23:11 ***
          Hide
          Knut Anders Hatlen added a comment -

          Couldn't we just wrap DRDAConnThread.run()'s call to closeSession() in a synchronized block (synchronizing on NetworkServerControlImpl.getShutdownSync()) and only call it if NetworkServerControlImpl.getShutdown() returns false? The server shutdown will take care of closing the sessions anyway.

          Show
          Knut Anders Hatlen added a comment - Couldn't we just wrap DRDAConnThread.run()'s call to closeSession() in a synchronized block (synchronizing on NetworkServerControlImpl.getShutdownSync()) and only call it if NetworkServerControlImpl.getShutdown() returns false? The server shutdown will take care of closing the sessions anyway.
          Hide
          Suresh Thalamati added a comment -

          This test is failing with a different error message also , might be related to the same issue:

                          • Diff file derbynetmats/DerbyNet/derbynetmats/DerbyNetAutoStart.diff
              • Start: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-08-25 11:00:42 ***
                1a2,4
                > java.sql.SQLException: No current connection.
                > agentError in Thread[DRDAConnThread_3,5,derby.daemons]
                > org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = database1; diagnostic msg = No current connection.
                Test Failed.
              • End: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-08-25 11:01:29 ***
          Show
          Suresh Thalamati added a comment - This test is failing with a different error message also , might be related to the same issue: Diff file derbynetmats/DerbyNet/derbynetmats/DerbyNetAutoStart.diff Start: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-08-25 11:00:42 *** 1a2,4 > java.sql.SQLException: No current connection. > agentError in Thread [DRDAConnThread_3,5,derby.daemons] > org.apache.derby.impl.drda.DRDAProtocolException: Execution failed because of Permanent Agent Error: SVRCOD = 40; RDBNAM = database1; diagnostic msg = No current connection. Test Failed. End: DerbyNetAutoStart jdk1.4.2 DerbyNet derbynetmats:derbynetmats 2006-08-25 11:01:29 ***
          Hide
          Fernanda Pizzorno added a comment -

          I have tested the solution using the rollbackAndClose() method (semantically equal to the forceClose() method discussed in DERBY-1020) and it actually does not help avoiding any of the exceptions seen in this issue from happening.

          The rollbackAndClose() method checks that the connection is open, rolls back and closes the connection synchronized on the connection. This does not prevent the exceptions from happening because the connection is being closed implicitly when the server shutdown stops the InternalDriver and/or the Database. Since the connection is being closed using its close method, the synchronization that was added does not prevent the connection from being closed by the shutdown (since that does not synchronize on the connection object).

          In my opinion there are two possible solutions for this issue: (1) change the test so that the set of exceptions will either not happen or does not show in the diff; (2) distinguish the exceptions that happen because the server was shut down from those happening for other reasons (Kathey Marsden has enumerated the scenarios in DERBY-1020), and ignore those that happen because the server was shut down.

          I think that (2) is a better solution, and I was wondering whether we can say that exceptions that happen during rollbackAndClose() can be considered as exceptions that happen due to a server shutdown iff (i)the connection is closed at the time the exception happens, and (ii)the session is being closed because the client left? For example, is there any other way to stop InternalDriver and/or the Database than a shutdown?

          Show
          Fernanda Pizzorno added a comment - I have tested the solution using the rollbackAndClose() method (semantically equal to the forceClose() method discussed in DERBY-1020 ) and it actually does not help avoiding any of the exceptions seen in this issue from happening. The rollbackAndClose() method checks that the connection is open, rolls back and closes the connection synchronized on the connection. This does not prevent the exceptions from happening because the connection is being closed implicitly when the server shutdown stops the InternalDriver and/or the Database. Since the connection is being closed using its close method, the synchronization that was added does not prevent the connection from being closed by the shutdown (since that does not synchronize on the connection object). In my opinion there are two possible solutions for this issue: (1) change the test so that the set of exceptions will either not happen or does not show in the diff; (2) distinguish the exceptions that happen because the server was shut down from those happening for other reasons (Kathey Marsden has enumerated the scenarios in DERBY-1020 ), and ignore those that happen because the server was shut down. I think that (2) is a better solution, and I was wondering whether we can say that exceptions that happen during rollbackAndClose() can be considered as exceptions that happen due to a server shutdown iff (i)the connection is closed at the time the exception happens, and (ii)the session is being closed because the client left? For example, is there any other way to stop InternalDriver and/or the Database than a shutdown?
          Hide
          Deepa Remesh added a comment -

          Thanks Fernanda for picking this up.

          I am wondering if this solution will fix all cases. I think this solution is in same lines as what was discussed in DERBY-1020 (use of forceClose() method) to do a rollback and close together. At that time itself, I was not convinced this solution will solve the problem. There are at least three different exceptions seen during shutdown in this test (stack traces are spread between DERBY-803, DERBY-273 and DERBY-1020). As seen from the stack traces, these exceptions can occur at calls to conn.rollback() or conn.close(). In a previous comment, Mike had posted another exception: "SQL Exception: No current connection. " The full stack trace for this exception is:
          java.sql.SQLException: No current connection.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:44)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:88)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:104)
          at org.apache.derby.impl.jdbc.Util.noCurrentConnection(Util.java:208)
          at org.apache.derby.impl.jdbc.EmbedConnection.checkIfClosed(EmbedConnection.java:1335)
          at org.apache.derby.impl.jdbc.EmbedConnection.setupContextStack(EmbedConnection.java:1513)
          at org.apache.derby.impl.jdbc.EmbedConnection.close(EmbedConnection.java:992)
          at org.apache.derby.impl.jdbc.EmbedConnection.close(EmbedConnection.java:970)
          at org.apache.derby.impl.drda.Database.close(Database.java:320)
          at org.apache.derby.impl.drda.Session.close(Session.java:110)
          at org.apache.derby.impl.drda.DRDAConnThread.closeSession(DRDAConnThread.java:7270)
          at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:269)

          This is similar to the exception I was getting when I was trying to simulate the failure by adding a sleep. When the network server thread was sleeping, the engine shutdown runs to completion and so I was getting "SQL Exception: No current connection. ".

          I think we can get these exceptions when we call conn.close() or conn.rollback() during shutdown of network server and in the meantime engine shutdown is:
          1) in progress - In this case, we get org.apache.derby.iapi.services.context.ShutdownException
          2) complete - In this case, we get java.sql.SQLException: No current connection.

          I may be mistaken here but just wanted to share my thoughts.

          Leaving my thoughts aside, if this patch solves the failure on your machine, then it may be okay to proceed with this approach. I have only a small comment about the changes in the patch. It may not be okay to remove this check "if ((conn != null) && !conn.isClosed())" before calling conn.rollbackAndClose() or conn.close() in Database.close(). I think we need to at least check the case that conn is not null.

          Show
          Deepa Remesh added a comment - Thanks Fernanda for picking this up. I am wondering if this solution will fix all cases. I think this solution is in same lines as what was discussed in DERBY-1020 (use of forceClose() method) to do a rollback and close together. At that time itself, I was not convinced this solution will solve the problem. There are at least three different exceptions seen during shutdown in this test (stack traces are spread between DERBY-803 , DERBY-273 and DERBY-1020 ). As seen from the stack traces, these exceptions can occur at calls to conn.rollback() or conn.close(). In a previous comment, Mike had posted another exception: "SQL Exception: No current connection. " The full stack trace for this exception is: java.sql.SQLException: No current connection. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:44) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:88) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:104) at org.apache.derby.impl.jdbc.Util.noCurrentConnection(Util.java:208) at org.apache.derby.impl.jdbc.EmbedConnection.checkIfClosed(EmbedConnection.java:1335) at org.apache.derby.impl.jdbc.EmbedConnection.setupContextStack(EmbedConnection.java:1513) at org.apache.derby.impl.jdbc.EmbedConnection.close(EmbedConnection.java:992) at org.apache.derby.impl.jdbc.EmbedConnection.close(EmbedConnection.java:970) at org.apache.derby.impl.drda.Database.close(Database.java:320) at org.apache.derby.impl.drda.Session.close(Session.java:110) at org.apache.derby.impl.drda.DRDAConnThread.closeSession(DRDAConnThread.java:7270) at org.apache.derby.impl.drda.DRDAConnThread.run(DRDAConnThread.java:269) This is similar to the exception I was getting when I was trying to simulate the failure by adding a sleep. When the network server thread was sleeping, the engine shutdown runs to completion and so I was getting "SQL Exception: No current connection. ". I think we can get these exceptions when we call conn.close() or conn.rollback() during shutdown of network server and in the meantime engine shutdown is: 1) in progress - In this case, we get org.apache.derby.iapi.services.context.ShutdownException 2) complete - In this case, we get java.sql.SQLException: No current connection. I may be mistaken here but just wanted to share my thoughts. Leaving my thoughts aside, if this patch solves the failure on your machine, then it may be okay to proceed with this approach. I have only a small comment about the changes in the patch. It may not be okay to remove this check "if ((conn != null) && !conn.isClosed())" before calling conn.rollbackAndClose() or conn.close() in Database.close(). I think we need to at least check the case that conn is not null.
          Hide
          Fernanda Pizzorno added a comment -

          I have managed to reproduce this failure, and I agree with Deepa's interpretation of what is happening. In the attached file (suggestion-803.diff), I have implemented an alternative solution to this problem than those suggested earlier. I have added the method rollbackAndClose() to EngineConnection. This method will check that the connection is opened, rollback and close the connection in a synchronized manner. This diff is not intended to be committed but I would appreciate if someone could comment the approach. Thank you!

          Show
          Fernanda Pizzorno added a comment - I have managed to reproduce this failure, and I agree with Deepa's interpretation of what is happening. In the attached file (suggestion-803.diff), I have implemented an alternative solution to this problem than those suggested earlier. I have added the method rollbackAndClose() to EngineConnection. This method will check that the connection is opened, rollback and close the connection in a synchronized manner. This diff is not intended to be committed but I would appreciate if someone could comment the approach. Thank you!
          Hide
          Deepa Remesh added a comment -

          Of late, I see this intermittent test failure is reported by few people. However, I have not been able to repro this issue on my machine. I am unassigning myself from this issue thinking someone else may be interested to pick it up.

          Here is what I have found along with a recap of the discussions so far on this issue and DERBY-1020:

          • This test uses derby.drda.startNetworkServer property. It checks that when embedded driver is loaded, it starts network server and when we do a system shutdown, it shuts down network server.
          • The shutdown of network server happens asynchronously. DRDAServerStarter calls NetworkServerControlImpl.directShutdown which simply notifies the waiting thread and returns. Thus the shutdown of network server happens in parallel with remaining shutdown processing in the engine.
          • Network server shutdown involves rollback and close of the EngineConnection object. Before calling these methods, it checks that the connection is not closed. From this, it seemed that this exception can occur only if the engine shutdown takes place in the meantime (after we check conn.isClosed() and when we are in conn.rollback())

          I think the exception occurs when the system shutdown happens in the meantime:

          if ((conn != null) && !conn.isClosed())
          {
          // --------------------------> Derby shutdown
          if (! forXA)

          { conn.rollback(); }
          conn.close();
          }


          The closest I have been able to simulating this error is only by introducing a sleep here:

          if ((conn != null) && !conn.isClosed())
          {
          // Introduce a sleep here ... just an attempt to simulate the failure
          try { Thread.sleep(5000); } catch (InterruptedException ie) { // }
          if(conn.isClosed()) { System.out.println("Connection has been closed in the meantime !!!"); }

          if (! forXA)
          { conn.rollback(); }

          conn.close();
          }

          If I run the test with this additional sleep, I always get "Connection has been closed in the meantime !!!" message and an exception in conn.rollback().

          • Irrespective of where the exception happens, the gist of the problem is what Kathey reported in DERBY-1020 (Network Server treats errors on cleanup of connections as an unexpected error after intentional shutdown of the database/server)
          • Possible solutions. These are just some ideas:

          1. Catch any SQLExceptions thrown in conn.rollback() and conn.close(). In the catch block, call conn.isClosed() to check if connection has been closed. If so, ignore the exception. However, this solution has drawback because we do not know if the connection is no longer active because of some unexpected event. Kathey made this comment in DERBY-1020
          "
          If an exception happens on rollback() or close() there are basically three scenarios
          .
          .
          .

          3) The connection is no longer active because of some unexpected event like a crash. There has been some debate in the past whether this type of exception needs to be logged. I thought session level exceptions in this block of code could safely be ignored since the fact that they were session severity meant the session was shutdown. Tomohito thought it was not good to ignore them and I understand that point.

          "
          Based on this, it may not be okay to just ignore exceptions.

          2. Use NetworkServerControlImpl.shutdown instead of NetworkServerControlImpl.directShutdown method. This is to ensure network server has done all the cleanup related to shutdown before we proceed with rest of shutdown processing in the engine. However, this will slow down the shutdown processing.

          3. Modify the test to redirect the exceptions during shutdown, similar to what is done in DERBY-273 for dataSourcePermissions_net test. This way, we can avoid seeing the intermittent failures in the regression test runs. This will not solve the real problem. Also, we may miss some real exceptions.

          Show
          Deepa Remesh added a comment - Of late, I see this intermittent test failure is reported by few people. However, I have not been able to repro this issue on my machine. I am unassigning myself from this issue thinking someone else may be interested to pick it up. Here is what I have found along with a recap of the discussions so far on this issue and DERBY-1020 : This test uses derby.drda.startNetworkServer property. It checks that when embedded driver is loaded, it starts network server and when we do a system shutdown, it shuts down network server. The shutdown of network server happens asynchronously. DRDAServerStarter calls NetworkServerControlImpl.directShutdown which simply notifies the waiting thread and returns. Thus the shutdown of network server happens in parallel with remaining shutdown processing in the engine. Network server shutdown involves rollback and close of the EngineConnection object. Before calling these methods, it checks that the connection is not closed. From this, it seemed that this exception can occur only if the engine shutdown takes place in the meantime (after we check conn.isClosed() and when we are in conn.rollback()) I think the exception occurs when the system shutdown happens in the meantime: if ((conn != null) && !conn.isClosed()) { // --------------------------> Derby shutdown if (! forXA) { conn.rollback(); } conn.close(); } The closest I have been able to simulating this error is only by introducing a sleep here: if ((conn != null) && !conn.isClosed()) { // Introduce a sleep here ... just an attempt to simulate the failure try { Thread.sleep(5000); } catch (InterruptedException ie) { // } if(conn.isClosed()) { System.out.println("Connection has been closed in the meantime !!!"); } if (! forXA) { conn.rollback(); } conn.close(); } If I run the test with this additional sleep, I always get "Connection has been closed in the meantime !!!" message and an exception in conn.rollback(). Irrespective of where the exception happens, the gist of the problem is what Kathey reported in DERBY-1020 (Network Server treats errors on cleanup of connections as an unexpected error after intentional shutdown of the database/server) Possible solutions. These are just some ideas: 1. Catch any SQLExceptions thrown in conn.rollback() and conn.close(). In the catch block, call conn.isClosed() to check if connection has been closed. If so, ignore the exception. However, this solution has drawback because we do not know if the connection is no longer active because of some unexpected event. Kathey made this comment in DERBY-1020 " If an exception happens on rollback() or close() there are basically three scenarios . . . 3) The connection is no longer active because of some unexpected event like a crash. There has been some debate in the past whether this type of exception needs to be logged. I thought session level exceptions in this block of code could safely be ignored since the fact that they were session severity meant the session was shutdown. Tomohito thought it was not good to ignore them and I understand that point. " Based on this, it may not be okay to just ignore exceptions. 2. Use NetworkServerControlImpl.shutdown instead of NetworkServerControlImpl.directShutdown method. This is to ensure network server has done all the cleanup related to shutdown before we proceed with rest of shutdown processing in the engine. However, this will slow down the shutdown processing. 3. Modify the test to redirect the exceptions during shutdown, similar to what is done in DERBY-273 for dataSourcePermissions_net test. This way, we can avoid seeing the intermittent failures in the regression test runs. This will not solve the real problem. Also, we may miss some real exceptions.
          Hide
          Mike Matrigali added a comment -

          I got the diff in a run against trunk - build 413780

          Failure Details:

                          • Diff file derbyall/derbynetclientmats/DerbyNetClient/derbynetmats/derbynetmats/DerbyNetAutoStart.diff
              • Start: DerbyNetAutoStart jdk1.5.0_02 DerbyNetClient derbynetmats:derbynetmats 2006-06-12 21:15:02 ***
                2a3,4
                > SQL Exception: No current connection.
                > agentThread[DRDAConnThread_7,5,derby.daemons]
                Test Failed.
              • End: DerbyNetAutoStart jdk1.5.0_02 DerbyNetClient derbynetmats:derbynetmats 2006-06-12 21:15:46 ***
          Show
          Mike Matrigali added a comment - I got the diff in a run against trunk - build 413780 Failure Details: Diff file derbyall/derbynetclientmats/DerbyNetClient/derbynetmats/derbynetmats/DerbyNetAutoStart.diff Start: DerbyNetAutoStart jdk1.5.0_02 DerbyNetClient derbynetmats:derbynetmats 2006-06-12 21:15:02 *** 2a3,4 > SQL Exception: No current connection. > agentThread [DRDAConnThread_7,5,derby.daemons] Test Failed. End: DerbyNetAutoStart jdk1.5.0_02 DerbyNetClient derbynetmats:derbynetmats 2006-06-12 21:15:46 ***
          Hide
          Deepa Remesh added a comment -

          I have not been able to repro this intermittent failure on my machine. I was using the stack traces attached to DERBY-273 to try to understand the scenario. I am thinking of following options to solve this problem:

          1. Solve the intermittent test failure by using the same solution as in DERBY-273. This will solve the test issue.
          2. Try to see if we can distinguish "expected" exceptions during shutdown and ignore them. However, I don't see a clear way of distinguishing this.
          3. I think the following thread explains the scenario:
          http://www.nabble.com/-jira--Commented%3A-%28DERBY-273%29-The-derbynet-dataSourcePermissions_net.java-test-fails-intermittently-p190292.html

          Scenario is: Server is shutdown. One of the threads finds that the client has disconnected. It tries to close the session but gets an exception during rollback because the shutdown has already taken place.

          The code where closeSession gets called is in DRDAConnThread.run method:

          } catch (Exception e) {
          if (e instanceof DRDAProtocolException &&
          ((DRDAProtocolException)e).isDisconnectException())

          { // client went away - this is O.K. here closeSession(); }

          else

          { handleException(e); }

          }

          In this scenario, since the client has already disconnected, can we ignore the exceptions when closing the session? Then, we can pass a flag to closeSession method asking to ignore exceptions if we are calling it when we receive a DisconnectException in the server. Any comments/suggestions ?

          Show
          Deepa Remesh added a comment - I have not been able to repro this intermittent failure on my machine. I was using the stack traces attached to DERBY-273 to try to understand the scenario. I am thinking of following options to solve this problem: 1. Solve the intermittent test failure by using the same solution as in DERBY-273 . This will solve the test issue. 2. Try to see if we can distinguish "expected" exceptions during shutdown and ignore them. However, I don't see a clear way of distinguishing this. 3. I think the following thread explains the scenario: http://www.nabble.com/-jira--Commented%3A-%28DERBY-273%29-The-derbynet-dataSourcePermissions_net.java-test-fails-intermittently-p190292.html Scenario is: Server is shutdown. One of the threads finds that the client has disconnected. It tries to close the session but gets an exception during rollback because the shutdown has already taken place. The code where closeSession gets called is in DRDAConnThread.run method: } catch (Exception e) { if (e instanceof DRDAProtocolException && ((DRDAProtocolException)e).isDisconnectException()) { // client went away - this is O.K. here closeSession(); } else { handleException(e); } } In this scenario, since the client has already disconnected, can we ignore the exceptions when closing the session? Then, we can pass a flag to closeSession method asking to ignore exceptions if we are calling it when we receive a DisconnectException in the server. Any comments/suggestions ?
          Hide
          Kathey Marsden added a comment -

          In this test an intentional shutdown causes the close of the connection by network server to encounter an error that prints to the console. It appears it may be the same issue as DERBY-273.

          Show
          Kathey Marsden added a comment - In this test an intentional shutdown causes the close of the connection by network server to encounter an error that prints to the console. It appears it may be the same issue as DERBY-273 .

            People

            • Assignee:
              Fernanda Pizzorno
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development