Derby
  1. Derby
  2. DERBY-5096

DisconnectException: "Connection was refused because the database DB was not found" when creating db for first time

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.7.1.1
    • Component/s: Network Server
    • Labels:
      None
    • Environment:
      System: OS/400 V7R1M0 on PowerPC; JVM: IBM J9 VM 2.4 (consistently reproducible).
      But also: Windows 2008 R2 on VMWare and Windows 7 64 bit with Sun VM (1.6) too (but not as consistently reproducible)
    • Issue & fix info:
      Patch Available
    • Bug behavior facts:
      Seen in production

      Description

      A process starts derby as network server (new NetworkServerControl, start) and waits for derby to respond (retry ping until success). After the server has started, the same process makes a connection with DriverManager.getConnection().

      The first time this process runs, this will always fail with "SQLNonTransientConnectionException: The connection was refused because the database Flows;create=true was not found." However, at the same time, "derby.log" and "Flows" directory are created in the derby home. The "Flows" directory contains "log", "seg0", "tmp", "db.lck", "service.properties", etc ...

      The second time the process is started, the connection will succeed and the "Flows" database can be used successfully. From now on, every start will be successful. After cleaning the derby home directory, you get the exception again after the first start.

      The attachment contains a Java program (DerbyGetConnection.java) that reproduces this problem consistently on our OS/400 V7R1M0 with IBM J9 VM. Probably this may happen on other systems too (Win7), but we were unable to reproduce it there (on my machine with Win7, this same program ran successful every run; might be a threading or I/O issue that occurs easily on slower hardware?).
      The attachment contains also derby tracing ("derby.drda.traceAll") and logging/tracing of the DriverManager.

      1. TestLowerCaseTraversal.java
        2 kB
        Kathey Marsden
      2. getconnection_reproduce.zip
        7 kB
        Steven Hendrickx
      3. derby-5096_uncondset_diff.txt
        2 kB
        Kathey Marsden
      4. derby-5096_diff.txt
        4 kB
        Kathey Marsden

        Issue Links

          Activity

          Steven Hendrickx created issue -
          Hide
          Steven Hendrickx added a comment -

          ZIP containing

          • DerbyGetConnection.java to reproduce problem
          • Derby tracing
          • JDBC/DriverManager trace
          Show
          Steven Hendrickx added a comment - ZIP containing DerbyGetConnection.java to reproduce problem Derby tracing JDBC/DriverManager trace
          Steven Hendrickx made changes -
          Field Original Value New Value
          Attachment getconnection_reproduce.zip [ 12472822 ]
          Hide
          Steven Hendrickx added a comment -

          Mind, since this is a SQLNonTransientConnectionException, we cannot retry to create the connection.

          Show
          Steven Hendrickx added a comment - Mind, since this is a SQLNonTransientConnectionException, we cannot retry to create the connection.
          Steven Hendrickx made changes -
          Description A process starts derby as network server (new NetworkServerControl, start) and waits for derby to respond (retry ping until success). After the server has started, the same process makes a connection with DriverManager.getConnection().

          The first time this process runs, this will always fail with "SQLNonTransientConnectionException: The connection was refused because the database Flows;create=true was not found." However, at the same time, "derby.log" and "Flows" directory are created in the derby home. The "Flows" directory contains "log", "seg0", "tmp", "db.lck", "service.properties", etc ...

          The second time the process is started, the connection will succeed and the "Flows" database can be used successfully.

          The attachment contains a Java program (DerbyGetConnection.java) that reproduces this problem consistently on our OS/400 V7R1M0 with IBM J9 VM. Probably this may happen on other systems too (Win7), but we were unable to reproduce it there (on my machine with Win7, this same program ran successful every run; might be a threading or I/O issue that occurs easily on slower hardware?).
          The attachment contains also derby tracing ("derby.drda.traceAll") and logging/tracing of the DriverManager.
          A process starts derby as network server (new NetworkServerControl, start) and waits for derby to respond (retry ping until success). After the server has started, the same process makes a connection with DriverManager.getConnection().

          The first time this process runs, this will always fail with "SQLNonTransientConnectionException: The connection was refused because the database Flows;create=true was not found." However, at the same time, "derby.log" and "Flows" directory are created in the derby home. The "Flows" directory contains "log", "seg0", "tmp", "db.lck", "service.properties", etc ...

          The second time the process is started, the connection will succeed and the "Flows" database can be used successfully. From now on, every start will be successful. After cleaning the derby home directory, you get the exception again after the first start.

          The attachment contains a Java program (DerbyGetConnection.java) that reproduces this problem consistently on our OS/400 V7R1M0 with IBM J9 VM. Probably this may happen on other systems too (Win7), but we were unable to reproduce it there (on my machine with Win7, this same program ran successful every run; might be a threading or I/O issue that occurs easily on slower hardware?).
          The attachment contains also derby tracing ("derby.drda.traceAll") and logging/tracing of the DriverManager.
          Steven Hendrickx made changes -
          Summary DisconnectException: "Connection was refused because the database DB was not found" on OS/400 DisconnectException: "Connection was refused because the database DB was not found" when creating db for first time
          Environment System: OS/400 V7R1M0 on PowerPC; JVM: IBM J9 VM 2.4 (continuously reproducible).
          Probably Windows 7 64 bit with Sun VM (1.6) too (but not reproduced)
          System: OS/400 V7R1M0 on PowerPC; JVM: IBM J9 VM 2.4 (consistently reproducible).
          But also: Windows 2008 R2 on VMWare and Windows 7 64 bit with Sun VM (1.6) too (but not as consistently reproducible)
          Hide
          Kathey Marsden added a comment -

          Hi Steven,

          On iseries I believe there is a java bug with getCanonicalPath() that can cause this problem. If you run the attached program TestLowerCaseTraversal.java on iseries, I think you will see that getCanonical path is not properly resolving the case in the parent directories

          $ java TestLowerCaseTraversal
          mkdirs to create dir with mixedcase string: test/NDee = true
          Original string: test/ndee/subdir
          cannonical path before mkdirs(), should be mixed case as that's what's on the machine: /home/MYRNA/tst/test/b11740/try4/test/ndee/subdir <<== Should be NDee!
          created the subdir? true
          cannonical path after mkdirs(): /home/MYRNA/tst/test/b11740/try4/test/NDee/subdir

          On Windows XP, I see the case properly resolved:

          mkdirs to create dir with mixedcase string: test/NDee = true
          Original string: test/ndee/subdir
          cannonical path before mkdirs(), should be mixed case as that's what's on the ma
          chine: C:\cygwin\home\kmarsden\repro\11740\test\NDee\subdir
          created the subdir? true
          cannonical path after mkdirs(): C:\cygwin\home\kmarsden\repro\11740\test\NDee\s
          ubdir

          The workaround on iseries is to use the matching case for the path of the database.

          Another situation where there is a problem with getCanonicalPath() on iseries that can cause this symptom is if there are multi-byte characters in the database name. (See DERBY-4149). I don't know a good work around for that one.

          I cannot speak to the vmware issues, but wonder if there is a casing or getCanonicalPath issue there too.

          Show
          Kathey Marsden added a comment - Hi Steven, On iseries I believe there is a java bug with getCanonicalPath() that can cause this problem. If you run the attached program TestLowerCaseTraversal.java on iseries, I think you will see that getCanonical path is not properly resolving the case in the parent directories $ java TestLowerCaseTraversal mkdirs to create dir with mixedcase string: test/NDee = true Original string: test/ndee/subdir cannonical path before mkdirs(), should be mixed case as that's what's on the machine: /home/MYRNA/tst/test/b11740/try4/test/ndee/subdir <<== Should be NDee! created the subdir? true cannonical path after mkdirs(): /home/MYRNA/tst/test/b11740/try4/test/NDee/subdir On Windows XP, I see the case properly resolved: mkdirs to create dir with mixedcase string: test/NDee = true Original string: test/ndee/subdir cannonical path before mkdirs(), should be mixed case as that's what's on the ma chine: C:\cygwin\home\kmarsden\repro\11740\test\NDee\subdir created the subdir? true cannonical path after mkdirs(): C:\cygwin\home\kmarsden\repro\11740\test\NDee\s ubdir The workaround on iseries is to use the matching case for the path of the database. Another situation where there is a problem with getCanonicalPath() on iseries that can cause this symptom is if there are multi-byte characters in the database name. (See DERBY-4149 ). I don't know a good work around for that one. I cannot speak to the vmware issues, but wonder if there is a casing or getCanonicalPath issue there too.
          Kathey Marsden made changes -
          Attachment TestLowerCaseTraversal.java [ 12473218 ]
          Kathey Marsden made changes -
          Link This issue is related to DERBY-4149 [ DERBY-4149 ]
          Hide
          Kathey Marsden added a comment -

          linking to DERBY-4149 as both are database not found that can be due to getCanonicalPath() problems.

          Show
          Kathey Marsden added a comment - linking to DERBY-4149 as both are database not found that can be due to getCanonicalPath() problems.
          Hide
          Steven Hendrickx added a comment -

          Kathey, you are correct: Basically our code does something like:

          new File(System.getProperty("user.home"), "productsubdir");

          On the OS/400 system, System.getProperty("user.home") returned "/home/SHENDRIC" for me, while this should be "/home/shendric" (ls in qsh or qp2term). This seems the root of all evil. When the code is changed to:

          new File(new File(System.getProperty("user.home")).getCanonicalFile, "productsubdir");

          the exception no longer occurs. Although this change seems to solve our problem, is there any plan to update Derby to anticipate this problem from within Derby itself?

          Anyway, thanks for the fast feed back and the great help.

          Show
          Steven Hendrickx added a comment - Kathey, you are correct: Basically our code does something like: new File(System.getProperty("user.home"), "productsubdir"); On the OS/400 system, System.getProperty("user.home") returned "/home/SHENDRIC" for me, while this should be "/home/shendric" (ls in qsh or qp2term). This seems the root of all evil. When the code is changed to: new File(new File(System.getProperty("user.home")).getCanonicalFile, "productsubdir"); the exception no longer occurs. Although this change seems to solve our problem, is there any plan to update Derby to anticipate this problem from within Derby itself? Anyway, thanks for the fast feed back and the great help.
          Hide
          Kathey Marsden added a comment - - edited

          It would be great to see a fix in the JVM's with the problem , so I think it would be worthwhile to report that you have hit the problem to the various JVM support organizations if you have support.

          I haven't looked into a possible workaround in Derby, but since it seems to affect not only OS/400 but also other platforms running on VMWare, it seems like it would be worthwhile to look into working around the problem in DERBY if possible. If you would like to work on that, I am sure members of the community can point you to the code area in question.

          Show
          Kathey Marsden added a comment - - edited It would be great to see a fix in the JVM's with the problem , so I think it would be worthwhile to report that you have hit the problem to the various JVM support organizations if you have support. I haven't looked into a possible workaround in Derby, but since it seems to affect not only OS/400 but also other platforms running on VMWare, it seems like it would be worthwhile to look into working around the problem in DERBY if possible. If you would like to work on that, I am sure members of the community can point you to the code area in question.
          Hide
          Myrna van Lunteren added a comment -

          I'm marking this as invalid as it's not really a problem in derby, the problem is in the jvm on iseries.
          If someone wants to implement a workaround within the derby code, they can reopen at that time.

          Show
          Myrna van Lunteren added a comment - I'm marking this as invalid as it's not really a problem in derby, the problem is in the jvm on iseries. If someone wants to implement a workaround within the derby code, they can reopen at that time.
          Myrna van Lunteren made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Invalid [ 6 ]
          Hide
          Kathey Marsden added a comment -

          Looking again at this issue, I think it is actually a usage issue, external to Derby and to getCanonicalPath.
          Here the user is using the System Property "user.home") which has CAPS
          /home/SHENDRIC where the actual directory does not.

          When the user called getCanonicalPath() on user home, e.g.

          , new File(new File(System.getProperty("user.home")).getCanonicalFile, "productsubdir");

          getCanonicalPath actually corrected the casing properly. I wonder if passed the whole string, if getCanonicalPath would resolve the whole thing properly.

          Also I would like to look at the actual exception and see what call is actually failing with the uppercase path and whether that should fail on iseries which generally ignores case.

          Show
          Kathey Marsden added a comment - Looking again at this issue, I think it is actually a usage issue, external to Derby and to getCanonicalPath. Here the user is using the System Property "user.home") which has CAPS /home/SHENDRIC where the actual directory does not. When the user called getCanonicalPath() on user home, e.g. , new File(new File(System.getProperty("user.home")).getCanonicalFile, "productsubdir"); getCanonicalPath actually corrected the casing properly. I wonder if passed the whole string, if getCanonicalPath would resolve the whole thing properly. Also I would like to look at the actual exception and see what call is actually failing with the uppercase path and whether that should fail on iseries which generally ignores case.
          Hide
          Kathey Marsden added a comment -

          I am going to see if we can find an internal workaround for this issue in Derby as it can be confusing to users.

          Show
          Kathey Marsden added a comment - I am going to see if we can find an internal workaround for this issue in Derby as it can be confusing to users.
          Kathey Marsden made changes -
          Resolution Invalid [ 6 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Kathey Marsden [ kmarsden ]
          Hide
          Kathey Marsden added a comment -

          I can get past the database creation issue if I change StroageFactoryService.createServiceRoot() to get the canonical path with:

          return serviceDirectory.getCanonicalPath();
          rather than
          return storageFactory.getCanonicalName();

          The real problem is that DirStorageFactory.java holds a canonicalName value which is cached from the very first call to getCanonicalPath() in its doInit() method which occurs before the directory is created, so has the wrong casing on iSeries. I think probably the best comprehensive solution would be to only cache that value once we have a call to getCanonicalName when the directory exists.

          Show
          Kathey Marsden added a comment - I can get past the database creation issue if I change StroageFactoryService.createServiceRoot() to get the canonical path with: return serviceDirectory.getCanonicalPath(); rather than return storageFactory.getCanonicalName(); The real problem is that DirStorageFactory.java holds a canonicalName value which is cached from the very first call to getCanonicalPath() in its doInit() method which occurs before the directory is created, so has the wrong casing on iSeries. I think probably the best comprehensive solution would be to only cache that value once we have a call to getCanonicalName when the directory exists.
          Hide
          Kathey Marsden added a comment -

          Here is a patch for this issue. It adjusts the canonicalName after mkdirs() if it does not match getCanonicalPath().

          I had considered have a fix in DirStorageFactory which would upon getting the canonicalPath search back and ensure the correct casing for the directories that existed, but opted for the other approach as it should not exist in situations where there is not a casing issue.

          Show
          Kathey Marsden added a comment - Here is a patch for this issue. It adjusts the canonicalName after mkdirs() if it does not match getCanonicalPath(). I had considered have a fix in DirStorageFactory which would upon getting the canonicalPath search back and ensure the correct casing for the directories that existed, but opted for the other approach as it should not exist in situations where there is not a casing issue.
          Kathey Marsden made changes -
          Attachment derby-5096_diff.txt [ 12504925 ]
          Hide
          Knut Anders Hatlen added a comment -

          + String serviceDirCanonicalPath = serviceDirectory.getCanonicalPath();
          + if (storageFactory.getCanonicalName() != serviceDirCanonicalPath)

          { + storageFactory.setCanonicalName(serviceDirCanonicalPath); + }

          I suppose we should have used the equals() method to compare the strings here. Or maybe just set the canonical name unconditionally (advantages: less code, and the new code path will also be tested on non-iSeries machines).

          Show
          Knut Anders Hatlen added a comment - + String serviceDirCanonicalPath = serviceDirectory.getCanonicalPath(); + if (storageFactory.getCanonicalName() != serviceDirCanonicalPath) { + storageFactory.setCanonicalName(serviceDirCanonicalPath); + } I suppose we should have used the equals() method to compare the strings here. Or maybe just set the canonical name unconditionally (advantages: less code, and the new code path will also be tested on non-iSeries machines).
          Hide
          Kathey Marsden added a comment -

          Thank you Knut. That is a good idea to make it unconditional as I recall one user reporting they also saw this intermittently on Windows vmware. This way the code path will be fully tested.

          attaching derby-5096_uncondset_diff.txt to do that.

          Show
          Kathey Marsden added a comment - Thank you Knut. That is a good idea to make it unconditional as I recall one user reporting they also saw this intermittently on Windows vmware. This way the code path will be fully tested. attaching derby-5096_uncondset_diff.txt to do that.
          Kathey Marsden made changes -
          Attachment derby-5096_uncondset_diff.txt [ 12505650 ]
          Kathey Marsden made changes -
          Issue & fix info Patch Available [ 10102 ]
          Hide
          Kathey Marsden added a comment -

          This is now checked into the 10.8 branch,10.7, and 10.5 as well.. Not closing as still need verification of the fix and to check into 10.6 and maybe go back even further.

          Show
          Kathey Marsden added a comment - This is now checked into the 10.8 branch,10.7, and 10.5 as well.. Not closing as still need verification of the fix and to check into 10.6 and maybe go back even further.
          Kathey Marsden made changes -
          Fix Version/s 10.5.3.2 [ 12315436 ]
          Fix Version/s 10.7.1.4 [ 12315902 ]
          Fix Version/s 10.8.2.3 [ 12318540 ]
          Fix Version/s 10.9.0.0 [ 12316344 ]
          Hide
          Kathey Marsden added a comment -

          ported fix back to all branches through 10.5. The 10.6 fix doesn't show up on subversion commits, but was checked in with revision 1212214

          Show
          Kathey Marsden added a comment - ported fix back to all branches through 10.5. The 10.6 fix doesn't show up on subversion commits, but was checked in with revision 1212214
          Kathey Marsden made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 10.6.2.3 [ 12315434 ]
          Resolution Fixed [ 1 ]
          Kathey Marsden made changes -
          Fix Version/s 10.8.3.0 [ 12323456 ]
          Fix Version/s 10.8.2.3 [ 12318540 ]
          Gavin made changes -
          Workflow jira [ 12600916 ] Default workflow, editable Closed status [ 12802143 ]
          Hide
          Knut Anders Hatlen added a comment -

          [bulk update: close all resolved issues that haven't had any activity the last year]

          Show
          Knut Anders Hatlen added a comment - [bulk update: close all resolved issues that haven't had any activity the last year]
          Knut Anders Hatlen made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Kathey Marsden
              Reporter:
              Steven Hendrickx
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development