Derby
  1. Derby
  2. DERBY-4733

Get error "Failed to create database - directory already exists" when attempting to connect to a database that failed during creation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 10.6.1.0
    • Fix Version/s: None
    • Component/s: Store
    • Environment:
      Windows XP
    • Urgency:
      Urgent
    • Bug behavior facts:
      Data corruption

      Description

      If you exit a process that is in the middle of creating a database, then when you try to access the same database again, you get the error "Failed to create database 'foo', see next exception" followed by "Directory xxx already exists".

      A stack trace is below. I am attaching a simple program that reproduces. Just try running it multiple times. I hit this after the second try.

      Opening database
      java.sql.SQLException: Failed to create database 'corruptme', see the next exception for details.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
      at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
      at java.sql.DriverManager.getConnection(Unknown Source)
      at java.sql.DriverManager.getConnection(Unknown Source)
      at CorruptMePlease$CreateTableTask.run(CorruptMePlease.java:34)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.sql.SQLException: Failed to create database 'corruptme', see the next exception for details.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
      ... 14 more
      Caused by: java.sql.SQLException: Directory C:\vontu\depot\sandbox_incremental\DerbyCorruption\corruptme already exists.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
      ... 11 more
      Caused by: ERROR XBM0J: Directory C:\vontu\depot\sandbox_incremental\DerbyCorruption\corruptme already exists.
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      at org.apache.derby.impl.services.monitor.StorageFactoryService$9.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
      at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
      ... 11 more

      1. CorruptMePlease.java
        2 kB
        David Van Couvering
      2. corruptme.zip
        7 kB
        David Van Couvering

        Issue Links

          Activity

          Hide
          David Van Couvering added a comment -

          Here is the simple program I used to recreate this issue

          Show
          David Van Couvering added a comment - Here is the simple program I used to recreate this issue
          Hide
          David Van Couvering added a comment -

          This is the database that can't be opened.

          Show
          David Van Couvering added a comment - This is the database that can't be opened.
          Hide
          David Van Couvering added a comment -

          Looking at this more closely, it appears that the error happens if the process exits while the database is booting for the first time (which means the database directory is being built up). Not sure how you could handle this correctly. Once the database is booted, then I can run multiple times and not have a problem.

          Show
          David Van Couvering added a comment - Looking at this more closely, it appears that the error happens if the process exits while the database is booting for the first time (which means the database directory is being built up). Not sure how you could handle this correctly. Once the database is booted, then I can run multiple times and not have a problem.
          Hide
          Mike Matrigali added a comment -

          I've changed the bug description to indicate the problem is about attempting to connect to a database which previously failed during creation. If anyone has better wording, feel free to improve it. I believe that the create table has nothing to do with the problem.

          This problem sounds medium hard to fix. Basically until the database is actually successfully created there is no existing mechanism to recover from errors while creating the database - especially if you are killing the process that is doing the database creation. There is a bootstrap phase where we are getting the database to the point that it can handle all errors, but until then we basically rely on not enabling access to the db until
          it is finished. The .zip you posted is missing the necessary bootstrap files that derby
          uses to open a database, and we don't lay down those files until we have finished
          creating the database. Also as a performance enhancement I believe that we
          optimize database creation by doing a lot of operations unlogged, as it is assumed
          no recovery is necessary for failures during middle of create database.

          As has been commented I can see severe problems with Derby going ahead and deleting
          the existing directory that it does not recognize as a database to make the creation
          succeed. It is too easy for the case to be a user error, rather than an attempt to connect
          to previously failed created database. With derby I think many people just leave
          the create=true flag on their connect line, so even though the database may think
          it is doing a create, it might be surprising to users. Probably what is causing this db
          to not boot is the missing bootstrap file service.properties, which I think is instantiated
          after we have forced the successful db creation to disk. Unfortunately I have seen
          too many users muck around with the internal files and trusting this file to not exist to
          mean that we can go ahead and delete existing seg0 and log seems like a way to
          cause trouble.

          It would seem useful if anyone is interested to provide better error messages. Step one could be just to change the text of this one to mention that it might be possible that
          a previous create database attempt failed and that user needs to clean up the existing
          directory. Step two could be to actually look for database artifacts that should get
          cleaned up, like log, seg0, tmp, db.lck.

          There may be smaller fixes that could be made if whatever if causing the db to not
          be created is something other than the process going away. I don't know if existing
          boot error handling is smart enough to know that it is failing during create database
          and if that context knows what objects it could delete as part of the error recovery.

          Show
          Mike Matrigali added a comment - I've changed the bug description to indicate the problem is about attempting to connect to a database which previously failed during creation. If anyone has better wording, feel free to improve it. I believe that the create table has nothing to do with the problem. This problem sounds medium hard to fix. Basically until the database is actually successfully created there is no existing mechanism to recover from errors while creating the database - especially if you are killing the process that is doing the database creation. There is a bootstrap phase where we are getting the database to the point that it can handle all errors, but until then we basically rely on not enabling access to the db until it is finished. The .zip you posted is missing the necessary bootstrap files that derby uses to open a database, and we don't lay down those files until we have finished creating the database. Also as a performance enhancement I believe that we optimize database creation by doing a lot of operations unlogged, as it is assumed no recovery is necessary for failures during middle of create database. As has been commented I can see severe problems with Derby going ahead and deleting the existing directory that it does not recognize as a database to make the creation succeed. It is too easy for the case to be a user error, rather than an attempt to connect to previously failed created database. With derby I think many people just leave the create=true flag on their connect line, so even though the database may think it is doing a create, it might be surprising to users. Probably what is causing this db to not boot is the missing bootstrap file service.properties, which I think is instantiated after we have forced the successful db creation to disk. Unfortunately I have seen too many users muck around with the internal files and trusting this file to not exist to mean that we can go ahead and delete existing seg0 and log seems like a way to cause trouble. It would seem useful if anyone is interested to provide better error messages. Step one could be just to change the text of this one to mention that it might be possible that a previous create database attempt failed and that user needs to clean up the existing directory. Step two could be to actually look for database artifacts that should get cleaned up, like log, seg0, tmp, db.lck. There may be smaller fixes that could be made if whatever if causing the db to not be created is something other than the process going away. I don't know if existing boot error handling is smart enough to know that it is failing during create database and if that context knows what objects it could delete as part of the error recovery.
          Hide
          Kathey Marsden added a comment -

          I agree it would be very bad to provide an option to remove an existing directory. Not only may a mistype blow away something totally unrelated, but sometimes this error occurs with corruption and people might get confused and blow away the evidence. Lastly I think such an option would be a potential security risk. I think Mike's suggestion of a error message change might be helpful.

          Show
          Kathey Marsden added a comment - I agree it would be very bad to provide an option to remove an existing directory. Not only may a mistype blow away something totally unrelated, but sometimes this error occurs with corruption and people might get confused and blow away the evidence. Lastly I think such an option would be a potential security risk. I think Mike's suggestion of a error message change might be helpful.
          Hide
          Rick Hillegas added a comment -

          Linking this issue to DERBY-4589 since they seem to be similar issues, perhaps identical. I think that we can improve the error message, as Mike suggests. I don't think we should take responsibility for deleting people's database directories.

          Show
          Rick Hillegas added a comment - Linking this issue to DERBY-4589 since they seem to be similar issues, perhaps identical. I think that we can improve the error message, as Mike suggests. I don't think we should take responsibility for deleting people's database directories.
          Hide
          Rick Hillegas added a comment -

          Resolving this issue as a duplicate of DERBY-4589.

          Show
          Rick Hillegas added a comment - Resolving this issue as a duplicate of DERBY-4589 .
          Hide
          Knut Anders Hatlen added a comment -

          [bulk update] Close all resolved issues that haven't been updated for more than one year.

          Show
          Knut Anders Hatlen added a comment - [bulk update] Close all resolved issues that haven't been updated for more than one year.

            People

            • Assignee:
              Unassigned
              Reporter:
              David Van Couvering
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development