Derby
  1. Derby
  2. DERBY-555

Unable to restart after disk is full

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 10.2.1.6
    • Fix Version/s: 10.1.3.3, 10.2.1.6
    • Component/s: Store
    • Labels:
      None
    • Environment:
      Sun Sparc Solaris, 1.4 JDK, Derby Client/Server

      Description

      1. Inserted data into the database until the disk was full. (5 clients inserting into 5 different tables in parallel.)
      2. Shut down the server
      3. Start the database again without freeing any disk space.

      When I try to start the database again, I get Null-pointer-exception, regardless of how I connect (have tried embedded, client server, ij, jdbc applications). I have not tried to free some space on the disk before starting.

      The call stack is not available right now (the computer I used had to be shut down due to problems with our cooling system), but the exception comes from the following line in RawStore.java:
      properties.put(Attribute.LOG_DEVICE, logFactory.getCanonicalLogPath());

      getCanonicalLogPath() returns null which results in a NPE in the hash table.
      A quick debug before the computer was stopped, showed that the logFactory was an instance of org.apache.derby.impl.store.raw.log.ReadOnly which always returns null in its getCanonicalLogPath().

      I suspect this may be related to the fact that I ran with the log in a non-default location.

      1. derby-555a.diff
        10 kB
        Øystein Grøvlen

        Issue Links

          Activity

          Hide
          Kathey Marsden added a comment -

          reopening to backport to 10.1. I need the read only boot message there and it will be good to have this fixed in 10.1 anyway.

          Show
          Kathey Marsden added a comment - reopening to backport to 10.1. I need the read only boot message there and it will be good to have this fixed in 10.1 anyway.
          Hide
          Kathey Marsden added a comment -

          I realize it is a bit late to ask this question now that the bug is closed, but is this fix appropriate to port to 10.1 for 10.1.2?
          It seems like something that would be great to have fixed for the release.

          Show
          Kathey Marsden added a comment - I realize it is a bit late to ask this question now that the bug is closed, but is this fix appropriate to port to 10.1 for 10.1.2? It seems like something that would be great to have fixed for the release.
          Hide
          Øystein Grøvlen added a comment -

          Committed by revision 325896

          Show
          Øystein Grøvlen added a comment - Committed by revision 325896
          Hide
          Øystein Grøvlen added a comment -

          Added a new patch that changes the message to derby.log and adds a test for this bug.

          With this patch, when a database is booted in read-only mode, the boot message in derby.log will contain information about that.

          I have also added a test, TurnsReadOnly.java, that without the fix, get the NPE. This is not in a disk full scenario, but it also occurs with a read-only DB directory. Since there is currently no way to turn write access back on from a Java program, the DB directory will be read-only after the test is run. Hence, the test framework will not be able to clean up the test directory. Therefore, this test is not added to the derbyall test suite.

          The following files are changed:

          M java/engine/org/apache/derby/impl/store/raw/RawStore.java
          Do not check log location for read-only databases (avoids NPE)
          M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
          Add information to boot message in derby.log when database is booted in read-only mode.
          M java/engine/org/apache/derby/iapi/reference/MessageId.java
          Added message id for the read-only part of the boot message
          M java/engine/org/apache/derby/loc/messages_en.properties
          Added text for the read-only part of the boot message
          M java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant
          Added TurnsReadOnly_app.properties
          A java/testing/org/apache/derbyTesting/functionTests/tests/store/TurnsReadOnly.java
          New test that boots a database with log in non-default location, shuts it down, sets DB directory to read-only, boots again and checks that DML is not allowed.
          A java/testing/org/apache/derbyTesting/functionTests/tests/store/TurnsReadOnly_app.properties
          Test properties
          A java/testing/org/apache/derbyTesting/functionTests/master/TurnsReadOnly.out
          Master file for new test.

          I am currently running derbyall and will update report when test is finished.

          Show
          Øystein Grøvlen added a comment - Added a new patch that changes the message to derby.log and adds a test for this bug. With this patch, when a database is booted in read-only mode, the boot message in derby.log will contain information about that. I have also added a test, TurnsReadOnly.java, that without the fix, get the NPE. This is not in a disk full scenario, but it also occurs with a read-only DB directory. Since there is currently no way to turn write access back on from a Java program, the DB directory will be read-only after the test is run. Hence, the test framework will not be able to clean up the test directory. Therefore, this test is not added to the derbyall test suite. The following files are changed: M java/engine/org/apache/derby/impl/store/raw/RawStore.java Do not check log location for read-only databases (avoids NPE) M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java Add information to boot message in derby.log when database is booted in read-only mode. M java/engine/org/apache/derby/iapi/reference/MessageId.java Added message id for the read-only part of the boot message M java/engine/org/apache/derby/loc/messages_en.properties Added text for the read-only part of the boot message M java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant Added TurnsReadOnly_app.properties A java/testing/org/apache/derbyTesting/functionTests/tests/store/TurnsReadOnly.java New test that boots a database with log in non-default location, shuts it down, sets DB directory to read-only, boots again and checks that DML is not allowed. A java/testing/org/apache/derbyTesting/functionTests/tests/store/TurnsReadOnly_app.properties Test properties A java/testing/org/apache/derbyTesting/functionTests/master/TurnsReadOnly.out Master file for new test. I am currently running derbyall and will update report when test is finished.
          Hide
          Øystein Grøvlen added a comment -

          If I understand you correctly, you suggest that I add an extra parameter to the boot message that can be used to indicate that a database is booted in read only mode. I can do that. I am a bit afraid that it will be less visible to someone inspecting the log file for the cause of the problem, but I can write READ-ONLY in upper case

          I have now reproduce the same bug by setting the DB directory to read-only. I will make a test based on that. Unfortunately, this cannot be added to derbyall before we can use J2SE 6 since there is currently no way to turn write access back on again. Hence, you will have to manually change the access rights in order to clean up the test directory after the test has been run.

          I will suggest a text to be added to the documentation. Should I create a separate JIRA issue for this?

          Show
          Øystein Grøvlen added a comment - If I understand you correctly, you suggest that I add an extra parameter to the boot message that can be used to indicate that a database is booted in read only mode. I can do that. I am a bit afraid that it will be less visible to someone inspecting the log file for the cause of the problem, but I can write READ-ONLY in upper case I have now reproduce the same bug by setting the DB directory to read-only. I will make a test based on that. Unfortunately, this cannot be added to derbyall before we can use J2SE 6 since there is currently no way to turn write access back on again. Hence, you will have to manually change the access rights in order to clean up the test directory after the test has been run. I will suggest a text to be added to the documentation. Should I create a separate JIRA issue for this?
          Hide
          Suresh Thalamati added a comment -

          Hi Øystein,

          Tested your patch , it looks good . I have couple of minor questions, related to the new error message :
          Database located at C:\suresht\databases\wombat has been booted in READ ONLY mod
          e. This will occur when the database is located in a jar file. This may also ha
          ppen when Derby is not able to write to the database directory; either because i
          t does not have the required access rights or because the disk is full.

          1) New Message added will be displayed on every boot from a read only media , Not sure if
          this will be annoying to the users who are booting derby from a read-only media ?

          2) Above Message added will never be displayed when booted from a JAR file. reference to jar in read-only mode ("... This will occur when the database is located in a jar file ...) may not be needed.

          How about modifying the general boot message to indicate that the database is booted in read-only mode ? and document the special case of disk full scenario some where ?

          I agree with you that there is no simple way to write a disk full test cases that can be added to nightly runs. But if you have a test case , I think it might be good idea to create special ad-hoc test directory and put it there , so that some one interested can run them manually at the time of a release.

          On a quick scan through the docs, found a section "Booting databases" section in the Developers guide("http://db.apache.org/derby/docs/10.1/devguide/) ; may be we can mention the disk full read-only boot special case there as a note.

          Thanks
          -suresht

          Show
          Suresh Thalamati added a comment - Hi Øystein, Tested your patch , it looks good . I have couple of minor questions, related to the new error message : Database located at C:\suresht\databases\wombat has been booted in READ ONLY mod e. This will occur when the database is located in a jar file. This may also ha ppen when Derby is not able to write to the database directory; either because i t does not have the required access rights or because the disk is full. 1) New Message added will be displayed on every boot from a read only media , Not sure if this will be annoying to the users who are booting derby from a read-only media ? 2) Above Message added will never be displayed when booted from a JAR file. reference to jar in read-only mode ("... This will occur when the database is located in a jar file ...) may not be needed. – How about modifying the general boot message to indicate that the database is booted in read-only mode ? and document the special case of disk full scenario some where ? I agree with you that there is no simple way to write a disk full test cases that can be added to nightly runs. But if you have a test case , I think it might be good idea to create special ad-hoc test directory and put it there , so that some one interested can run them manually at the time of a release. On a quick scan through the docs, found a section "Booting databases" section in the Developers guide("http://db.apache.org/derby/docs/10.1/devguide/) ; may be we can mention the disk full read-only boot special case there as a note. Thanks -suresht
          Hide
          Øystein Grøvlen added a comment -

          I have attached a patch that aims to resolve this issue as follows:

          • If database is set to read-only, print a message in derby.log about this.
          • If the database is read-only, do not worry about log path. This avoids the null pointer exception.

          The following files are changed:
          M java/engine/org/apache/derby/impl/store/raw/RawStore.java
          M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
          M java/engine/org/apache/derby/iapi/reference/MessageId.java
          M java/engine/org/apache/derby/loc/messages_en.properties

          If this fix is applied, the database will boot in read-only mode if there is not free disk space in the database directory. A message of this will then be printed to derby.log. This is far from an ideal solution since an application that experience this will not get any warning/error when connecting. However, if it laters tries to modify data, it will get exceptions like "ERROR 25502: An SQL data change is not permitted for a read-only connection, user or database". I think many Derby users will be confused by this message since I am not sure the aspects of read-only connections/users/databases are well known to most Derby users.

          I guess the documentation should be updated to inform about this, but I am not quite sure where to put it. Any suggestions?

          I have not made any tests for this. Do people think it is necessary? It is a bit difficult to make a standard test that fills disk devices. I could experiment to see if I can get the same NPE problem with a read-only database.

          Show
          Øystein Grøvlen added a comment - I have attached a patch that aims to resolve this issue as follows: If database is set to read-only, print a message in derby.log about this. If the database is read-only, do not worry about log path. This avoids the null pointer exception. The following files are changed: M java/engine/org/apache/derby/impl/store/raw/RawStore.java M java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java M java/engine/org/apache/derby/iapi/reference/MessageId.java M java/engine/org/apache/derby/loc/messages_en.properties If this fix is applied, the database will boot in read-only mode if there is not free disk space in the database directory. A message of this will then be printed to derby.log. This is far from an ideal solution since an application that experience this will not get any warning/error when connecting. However, if it laters tries to modify data, it will get exceptions like "ERROR 25502: An SQL data change is not permitted for a read-only connection, user or database". I think many Derby users will be confused by this message since I am not sure the aspects of read-only connections/users/databases are well known to most Derby users. I guess the documentation should be updated to inform about this, but I am not quite sure where to put it. Any suggestions? I have not made any tests for this. Do people think it is necessary? It is a bit difficult to make a standard test that fills disk devices. I could experiment to see if I can get the same NPE problem with a read-only database.
          Hide
          Øystein Grøvlen added a comment -

          Since it seems not to be possible to detect full disk in Java, my suggestion is to write a message to derby.log when a database is made read-only which states that reason may either be because it is lacking write access or because the disk is full.

          Show
          Øystein Grøvlen added a comment - Since it seems not to be possible to detect full disk in Java, my suggestion is to write a message to derby.log when a database is made read-only which states that reason may either be because it is lacking write access or because the disk is full.
          Hide
          Øystein Grøvlen added a comment -

          The log disk was not full. Neither was the disk where the derby.log file is written.

          Show
          Øystein Grøvlen added a comment - The log disk was not full. Neither was the disk where the derby.log file is written.
          Hide
          Francois Orsini added a comment -

          There is no DiskFullException (working) exception in the current JVM - there are a couple of JSR's (i.e. JSR-203) related to this as well as getting access to some extra FS api but it is now planned for jdk 1.6 or possibly later (been requested for years)

          There is a potential DiskFullException exception workaronud posted below:
          http://weblog.janek.org/Archive/2004/12/20/ExceptionWhenWritingToAFu.html
          (there is also an interesting behavior as far as file creation on a disk full - kinda skeptical about that but eh)

          There is also no API to return how space is left on a disk (device) - this is part of the related JSR's as well...

          --francois

          Show
          Francois Orsini added a comment - There is no DiskFullException (working) exception in the current JVM - there are a couple of JSR's (i.e. JSR-203) related to this as well as getting access to some extra FS api but it is now planned for jdk 1.6 or possibly later (been requested for years) There is a potential DiskFullException exception workaronud posted below: http://weblog.janek.org/Archive/2004/12/20/ExceptionWhenWritingToAFu.html (there is also an interesting behavior as far as file creation on a disk full - kinda skeptical about that but eh) There is also no API to return how space is left on a disk (device) - this is part of the related JSR's as well... --francois
          Hide
          Suresh Thalamati added a comment -

          One case I can think of database disk is full and system is unable to create even a lock (db.lck) file also. In derby , there is this logic that declares database is read only , when it is not able to create files. Idea is to boot database in jar files ..etc in read only mode . If file creation IO error happens to occur , before it load the log factory is loaded , then it boots the read only verion of the log factory implementation, thinking that databse is read only.

          you may want to look at the following code to see , if this is what is happening in u'r case:
          BaseDataFileFatory.java:

          1)
          private void bootLogFactory(boolean create, Properties startParams) throws StandardException

          { if (isReadOnly()) startParams.put(LogFactory.RUNTIME_ATTRIBUTES, LogFactory.RT_READONLY); logFactory = (LogFactory) Monitor.bootServiceModule(create, this, rawStoreFactory.getLogFactoryModule(), startParams); }

          2) boot() method

          3) ,private void privGetJBMSLockOnDB() throws StandardException this method marks the database as readonly , if it is not able to create db.lck files.

          Marking database on disk full case seems to be incorrect as u mentioned. Problem here is ,we want to automatically figure out the read only databases, but java does not seem have a DiskFull exception!!

          Thanks
          -suresht

          Show
          Suresh Thalamati added a comment - One case I can think of database disk is full and system is unable to create even a lock (db.lck) file also. In derby , there is this logic that declares database is read only , when it is not able to create files. Idea is to boot database in jar files ..etc in read only mode . If file creation IO error happens to occur , before it load the log factory is loaded , then it boots the read only verion of the log factory implementation, thinking that databse is read only. you may want to look at the following code to see , if this is what is happening in u'r case: BaseDataFileFatory.java: 1) private void bootLogFactory(boolean create, Properties startParams) throws StandardException { if (isReadOnly()) startParams.put(LogFactory.RUNTIME_ATTRIBUTES, LogFactory.RT_READONLY); logFactory = (LogFactory) Monitor.bootServiceModule(create, this, rawStoreFactory.getLogFactoryModule(), startParams); } 2) boot() method 3) ,private void privGetJBMSLockOnDB() throws StandardException this method marks the database as readonly , if it is not able to create db.lck files. – Marking database on disk full case seems to be incorrect as u mentioned. Problem here is ,we want to automatically figure out the read only databases, but java does not seem have a DiskFull exception!! Thanks -suresht
          Hide
          Øystein Grøvlen added a comment -

          Can anyone explain why LogFactory becomes an instance of ReadOnly when I restart the database? I have not been able to figure out how/where/when that happens.

          Show
          Øystein Grøvlen added a comment - Can anyone explain why LogFactory becomes an instance of ReadOnly when I restart the database? I have not been able to figure out how/where/when that happens.

            People

            • Assignee:
              Øystein Grøvlen
              Reporter:
              Øystein Grøvlen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development