Derby
  1. Derby
  2. DERBY-646

In-memory backend storage support

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.5.1.1
    • Fix Version/s: 10.5.1.1
    • Component/s: Store
    • Labels:
      None
    • Environment:
      All

      Description

      To allow creation and modification of databases in-memory without requiring disk access or space to store the database.

      1. derby-646-1a-raw-compiles.diff
        83 kB
        Kristian Waagan
      2. derby-646-1a-raw-compiles.stat
        1 kB
        Kristian Waagan
      3. derby-646-20090222.diff
        196 kB
        Cheng Che Chen
      4. derby-646-20090222.stat
        2 kB
        Cheng Che Chen
      5. derby-646-2a-vfmem_first_rev.diff
        87 kB
        Kristian Waagan
      6. derby-646-2a-vfmem_first_rev.stat
        1 kB
        Kristian Waagan
      7. derby-646-2b-vfmem_first_rev.diff
        95 kB
        Kristian Waagan
      8. derby-646-3a-jmx_experiment.diff
        5 kB
        Kristian Waagan
      9. derby-646-performance_comparison_1a.txt
        3 kB
        Kristian Waagan
      10. svn.diff
        83 kB
        Stephen Fitch

        Issue Links

          Activity

          Hide
          Stephen Fitch added a comment -

          I currently have an in-memory implementation of the storage factory interface "working" with the embedded client. It requires a very minimal change to one existing class and the addition of 6 more.

          It's easily plugged into the embeded client with -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory, but I'm still trying to work out how to get it going with the network server. (use jdbc:derby:memory:<database;attributes> as connection url)

          MemoryStorageFactory - In-memory implementation of StorageFactory

          MemoryStorageFile extends InputStreamFile - wraps most calls to another class with a hashtable keyed by full filepath names

          MemoryStorageRandomAccessFile - extends java.io.RandomAccessFile and overrides its' read, write and native methods.

          DynamicByteArrayIOStream - Readable, writable dynamically accessible byte array. This was done because I didnt want to have to implement the DataInput and DataOutput interfaces specified by the StorageRandomAccessFile interface (which would have been just copying them from java.io.RandomAccessFile). It is by no means an optimal implementation, but it works.

          DynamicByteArrayIOStreamFile - the class above extended with file attributes (ex: do I exist, am I a directory).

          MemoryMetadata - generalized location for StorageFile paths and DynamicByteArrayIOStreamFile objects. Also

          The naming convention is debatable and as this is still a work in progress can be changed.

          Any and all suggestions welcome

          Show
          Stephen Fitch added a comment - I currently have an in-memory implementation of the storage factory interface "working" with the embedded client. It requires a very minimal change to one existing class and the addition of 6 more. It's easily plugged into the embeded client with -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory, but I'm still trying to work out how to get it going with the network server. (use jdbc:derby:memory:<database;attributes> as connection url) MemoryStorageFactory - In-memory implementation of StorageFactory MemoryStorageFile extends InputStreamFile - wraps most calls to another class with a hashtable keyed by full filepath names MemoryStorageRandomAccessFile - extends java.io.RandomAccessFile and overrides its' read, write and native methods. DynamicByteArrayIOStream - Readable, writable dynamically accessible byte array. This was done because I didnt want to have to implement the DataInput and DataOutput interfaces specified by the StorageRandomAccessFile interface (which would have been just copying them from java.io.RandomAccessFile). It is by no means an optimal implementation, but it works. DynamicByteArrayIOStreamFile - the class above extended with file attributes (ex: do I exist, am I a directory). MemoryMetadata - generalized location for StorageFile paths and DynamicByteArrayIOStreamFile objects. Also The naming convention is debatable and as this is still a work in progress can be changed. Any and all suggestions welcome
          Hide
          Mike Perham added a comment -

          I would love to see Derby in-memory only support for unit testing purposes. This would solve a major problem of ours.

          Show
          Mike Perham added a comment - I would love to see Derby in-memory only support for unit testing purposes. This would solve a major problem of ours.
          Hide
          Stephen Fitch added a comment -

          Implementation rewritten to be more modular. Requires no changes to existing code to work. Define property derby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory and use connection url jdbc:derby:memory:databasename

          Show
          Stephen Fitch added a comment - Implementation rewritten to be more modular. Requires no changes to existing code to work. Define property derby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory and use connection url jdbc:derby:memory:databasename
          Hide
          James Synge added a comment -

          I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great.

          Show
          James Synge added a comment - I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great.
          Hide
          Daniel John Debrunner added a comment -

          I think it would be great to have an in-memory db support like this and thanks for working on it Stephen,
          but there are a few issues to resolve first.

          We would need an ICLA on file at apache from Stephen Fitch before this could be committed into Derby.

          A quick look at the patch showed a couple of extra classes in the org.apache.derby.io package which is the published api for
          the virtual file system, classes such as VirtualFileData and VirtualFS. I can't tell what the purpose of these classes are since there
          are no comments for them, thus it makes it impossible for anyone to implement them without guessing what the requirements are.

          Also seemed to be no tests provided with the patch, so how would a reviewer know that this works or not?

          Copyrights on all the files seem to have incorrect dates, and file names.

          The wiki has information on what is required:

          http://wiki.apache.org/db-derby/DerbyContributorChecklist

          http://wiki.apache.org/db-derby/DerbyCommitProcess

          http://wiki.apache.org/db-derby/PatchAdvice

          Show
          Daniel John Debrunner added a comment - I think it would be great to have an in-memory db support like this and thanks for working on it Stephen, but there are a few issues to resolve first. We would need an ICLA on file at apache from Stephen Fitch before this could be committed into Derby. A quick look at the patch showed a couple of extra classes in the org.apache.derby.io package which is the published api for the virtual file system, classes such as VirtualFileData and VirtualFS. I can't tell what the purpose of these classes are since there are no comments for them, thus it makes it impossible for anyone to implement them without guessing what the requirements are. Also seemed to be no tests provided with the patch, so how would a reviewer know that this works or not? Copyrights on all the files seem to have incorrect dates, and file names. The wiki has information on what is required: http://wiki.apache.org/db-derby/DerbyContributorChecklist http://wiki.apache.org/db-derby/DerbyCommitProcess http://wiki.apache.org/db-derby/PatchAdvice
          Hide
          Ashwin Jayaprakash added a comment -

          Hi,
          I would like to try this patch. But, I don't see any JAR or Java files attached. Could you pls tell me where I can download this patch?

          Thanks,
          Ashwin.

          Show
          Ashwin Jayaprakash added a comment - Hi, I would like to try this patch. But, I don't see any JAR or Java files attached. Could you pls tell me where I can download this patch? Thanks, Ashwin.
          Hide
          Andrew McIntyre added a comment -

          The patch is attached to this issue as a diff file. You need to get a local copy of the Derby source, then download the patch from JIRA (generally, right-click -> save as... on the attachment 'svn.diff'), then patch your local copy using 'patch -p0 < svn.diff' or your IDE, if it supports patching. I know Eclipse will apply patches generated by CVS and SVN. You will then need to build a local copy of Derby with the patch applied to try out the patch.

          For help checking out the Derby source code from Subversion, see: http://db.apache.org/derby/derby_downloads.html#Derby+source+code

          For help building, see BUILDING.txt at the top of the source tree and/or http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt

          Show
          Andrew McIntyre added a comment - The patch is attached to this issue as a diff file. You need to get a local copy of the Derby source, then download the patch from JIRA (generally, right-click -> save as... on the attachment 'svn.diff'), then patch your local copy using 'patch -p0 < svn.diff' or your IDE, if it supports patching. I know Eclipse will apply patches generated by CVS and SVN. You will then need to build a local copy of Derby with the patch applied to try out the patch. For help checking out the Derby source code from Subversion, see: http://db.apache.org/derby/derby_downloads.html#Derby+source+code For help building, see BUILDING.txt at the top of the source tree and/or http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt
          Hide
          David Van Couvering added a comment -

          Are separate tests needed for this feature. Isn't this a configuration option, e.g. couldn't we run the storemats suite with the system configured to use this store module?

          Show
          David Van Couvering added a comment - Are separate tests needed for this feature. Isn't this a configuration option, e.g. couldn't we run the storemats suite with the system configured to use this store module?
          Hide
          Daniel John Debrunner added a comment -

          Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration.
          As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for.
          Why would a release not be needed?

          Show
          Daniel John Debrunner added a comment - Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration. As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for. Why would a release not be needed?
          Hide
          Stefan Arentz added a comment -

          Has anything happened with this patch? I don't see any comments from the original submitter.

          Show
          Stefan Arentz added a comment - Has anything happened with this patch? I don't see any comments from the original submitter.
          Hide
          Kristian Waagan added a comment -

          I tried to apply this patch. It applied (some chunks needed to be offset), but it did not compile.
          I'm clearing the patch available flag, as the patch is clearly not ready for commit.

          There seems to be some interest for this feature, so I might spend some time to wash the patch to the level where it can be applied and tested. However, some more work from the original contributer would be greatly appreciated For instance, documenting new classes would be very helpful (as requested earlier by Dan).

          Note that I'm also unassigning the original contributer for now, since it has been approx 8 months since the last activity.
          Feel free to reclaim the issue if you plan working on it!

          Show
          Kristian Waagan added a comment - I tried to apply this patch. It applied (some chunks needed to be offset), but it did not compile. I'm clearing the patch available flag, as the patch is clearly not ready for commit. There seems to be some interest for this feature, so I might spend some time to wash the patch to the level where it can be applied and tested. However, some more work from the original contributer would be greatly appreciated For instance, documenting new classes would be very helpful (as requested earlier by Dan). Note that I'm also unassigning the original contributer for now, since it has been approx 8 months since the last activity. Feel free to reclaim the issue if you plan working on it!
          Hide
          Kristian Waagan added a comment -

          'derby-646-1a-raw-compiles.diff' is a minor fix of the original patch 'svn.diff'. The patch now applies to trunk and compiles. No other clean-ups/changes have been done, but I had to rewrite a little bit to solve circular compile dependencies.

          As far as I can tell, the basic functionality is working. I was able to run with the memory storage backend, which creates a database directory with lock files and a tmp directory. Tried running a few tests with a "hacked" JDBCClient implementation, and the tests passed. Had some problems with warnings regarding dual boot (missing shutdown/clean-up?).

          I think a little more clean-up should be done first (implement a missing method - getURL, remove commented out code, add some more comments), and then the patch can be reviewed for functional/design improvements.
          We might want some tests written specifically for the memory backend, and then run the existing tests with the memory backend as well.

          Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?

          Show
          Kristian Waagan added a comment - 'derby-646-1a-raw-compiles.diff' is a minor fix of the original patch 'svn.diff'. The patch now applies to trunk and compiles. No other clean-ups/changes have been done, but I had to rewrite a little bit to solve circular compile dependencies. As far as I can tell, the basic functionality is working. I was able to run with the memory storage backend, which creates a database directory with lock files and a tmp directory. Tried running a few tests with a "hacked" JDBCClient implementation, and the tests passed. Had some problems with warnings regarding dual boot (missing shutdown/clean-up?). I think a little more clean-up should be done first (implement a missing method - getURL, remove commented out code, add some more comments), and then the patch can be reviewed for functional/design improvements. We might want some tests written specifically for the memory backend, and then run the existing tests with the memory backend as well. Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?
          Hide
          Andrew McIntyre added a comment -

          "Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?"

          The answer to this question is yes, sorry I missed it when it was posed several months ago. The 'Grant license to ASF' box was also checked when the original code was submitted, so anyone who wants to work on getting this code complete enough to be committed should feel free to do so.

          Show
          Andrew McIntyre added a comment - "Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?" The answer to this question is yes, sorry I missed it when it was posed several months ago. The 'Grant license to ASF' box was also checked when the original code was submitted, so anyone who wants to work on getting this code complete enough to be committed should feel free to do so.
          Hide
          Cliff Evans added a comment -

          I think people are under valuing this request as a useful tool in production code. I'm currently using H2 to implement a multi-threaded matching engine which takes a series of events and matches aggregated information held in an in-memory database in real time. The datasets never get too large but the matching can be quite complex. Being able to use the expressive power of SQL without the overhead of pesisting the information to disk makes the matching process very fast while providing all of the thread synchronisation required in a tried and tested package. My only problem with H2 is the granularity of its locking. Table or nothing. This causes blocking as the tables are updated which an in-memory, row-locked version of Derby would reduce considerably. We're talking manna from heaven here.

          Are there any plans to include this in the main release?

          Show
          Cliff Evans added a comment - I think people are under valuing this request as a useful tool in production code. I'm currently using H2 to implement a multi-threaded matching engine which takes a series of events and matches aggregated information held in an in-memory database in real time. The datasets never get too large but the matching can be quite complex. Being able to use the expressive power of SQL without the overhead of pesisting the information to disk makes the matching process very fast while providing all of the thread synchronisation required in a tried and tested package. My only problem with H2 is the granularity of its locking. Table or nothing. This causes blocking as the tables are updated which an in-memory, row-locked version of Derby would reduce considerably. We're talking manna from heaven here. Are there any plans to include this in the main release?
          Hide
          Myrna van Lunteren added a comment -

          Although this issue has had a number of votes, I have to say, it currently doesn't look like it'll make it for 10.3 - it appears there has not been any testing of the patch by prospecting users nor did anyone sign up to drive this to completion.

          If you are interested in this, feel free to give the patch a try and help driving this into the next release after 10.3.

          Show
          Myrna van Lunteren added a comment - Although this issue has had a number of votes, I have to say, it currently doesn't look like it'll make it for 10.3 - it appears there has not been any testing of the patch by prospecting users nor did anyone sign up to drive this to completion. If you are interested in this, feel free to give the patch a try and help driving this into the next release after 10.3.
          Hide
          Kathey Marsden added a comment -

          Knut Magne Solem submitted a proof of concept for this issue is attached to DERBY-2798

          Show
          Kathey Marsden added a comment - Knut Magne Solem submitted a proof of concept for this issue is attached to DERBY-2798
          Hide
          Dave Syer added a comment -

          I would like to test this storage strategy, but I don't know how to set it up. I have compiled the code against 10.2.1.6, and now I want to obtain a connection (ideally through a DataSource). This is what I tried

          System.setProperty("derby.subSubProtocol.memory",
          "org.apache.derby.impl.io.MemoryStorageFactory");

          EmbeddedDataSource ds = new EmbeddedDataSource();
          ds.setDatabaseName("memory:test");
          ds.setCreateDatabase("create");
          Connection connection = ds.getConnection();

          It fails with

          java.sql.SQLException: Failed to start database 'memory:test', see the next exception for details.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
          ...

          There is no "next exception" in the stack trace.

          I also tried:

          System.setProperty("derby.subSubProtocol.memory",
          "org.apache.derby.impl.io.MemoryStorageFactory");
          DriverManager.registerDriver(new org.apache.derby.jdbc.EmbeddedDriver());
          Connection connection = DriverManager.getConnection("jdbc:derby:memory:test")

          which fails the same way.

          Show
          Dave Syer added a comment - I would like to test this storage strategy, but I don't know how to set it up. I have compiled the code against 10.2.1.6, and now I want to obtain a connection (ideally through a DataSource). This is what I tried System.setProperty("derby.subSubProtocol.memory", "org.apache.derby.impl.io.MemoryStorageFactory"); EmbeddedDataSource ds = new EmbeddedDataSource(); ds.setDatabaseName("memory:test"); ds.setCreateDatabase("create"); Connection connection = ds.getConnection(); It fails with java.sql.SQLException: Failed to start database 'memory:test', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) ... There is no "next exception" in the stack trace. I also tried: System.setProperty("derby.subSubProtocol.memory", "org.apache.derby.impl.io.MemoryStorageFactory"); DriverManager.registerDriver(new org.apache.derby.jdbc.EmbeddedDriver()); Connection connection = DriverManager.getConnection("jdbc:derby:memory:test") which fails the same way.
          Hide
          Dave Syer added a comment -

          I trie dit with 10.3.1.2 in case that was the problem. Had to patch the patch slightly to implement two methods that were unimplemented abstract (guessed the implementation, but did get far enough to test it).

          Now I get a more helpful stack trace (but maybe this is another problem?):

          Show
          Dave Syer added a comment - I trie dit with 10.3.1.2 in case that was the problem. Had to patch the patch slightly to implement two methods that were unimplemented abstract (guessed the implementation, but did get far enough to test it). Now I get a more helpful stack trace (but maybe this is another problem?):
          Hide
          Dave Syer added a comment -

          java.lang.SecurityException: sealing violation: can't seal package org.apache.derby.impl.io: already loaded
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:235)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
          at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:164)
          at test.EmeddedMemoryTests.setUp(EmeddedMemoryTests.java:41)
          at junit.framework.TestCase.runBare(TestCase.java:125)
          at junit.framework.TestResult$1.protect(TestResult.java:106)
          at junit.framework.TestResult.runProtected(TestResult.java:124)
          at junit.framework.TestResult.run(TestResult.java:109)
          at junit.framework.TestCase.run(TestCase.java:118)
          at junit.framework.TestSuite.runTest(TestSuite.java:208)
          at junit.framework.TestSuite.run(TestSuite.java:203)
          at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
          at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
          at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
          at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
          at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
          at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)

          Show
          Dave Syer added a comment - java.lang.SecurityException: sealing violation: can't seal package org.apache.derby.impl.io: already loaded at java.net.URLClassLoader.defineClass(URLClassLoader.java:235) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:164) at test.EmeddedMemoryTests.setUp(EmeddedMemoryTests.java:41) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
          Hide
          Ashwin Jayaprakash added a comment -

          The Derby JAR file is sealed - and this means that all the Derby packages specified in the JAR file's Manifest are sealed and you cannot add a new Class that shares the Package names to the Classpath.

          Since there are classes in org.apache.derby.impl.io Package in the Derby "sealed" JAR, you cannot append a Class like org.apache.derby.impl.io.PureMemStore that resides outside the JAR to the ClassPath. Unless I'm completely wrong, you will have to use the Source code and add your files there and then recreate the JAR file.

          Show
          Ashwin Jayaprakash added a comment - The Derby JAR file is sealed - and this means that all the Derby packages specified in the JAR file's Manifest are sealed and you cannot add a new Class that shares the Package names to the Classpath. Since there are classes in org.apache.derby.impl.io Package in the Derby "sealed" JAR, you cannot append a Class like org.apache.derby.impl.io.PureMemStore that resides outside the JAR to the ClassPath. Unless I'm completely wrong, you will have to use the Source code and add your files there and then recreate the JAR file.
          Hide
          Dave Syer added a comment -

          Is there an "unsealed" derby.jar available? I guess I just lost interest in testing this very interesting looking feature. I hope someone else can do it.

          Show
          Dave Syer added a comment - Is there an "unsealed" derby.jar available? I guess I just lost interest in testing this very interesting looking feature. I hope someone else can do it.
          Hide
          Ashwin Jayaprakash added a comment -

          You can un-JAR the file to a directory, paste your compiled Classes into that folder and use this "exploded" directory with the original classes and your changes in the Classpath instead of having the JAR in the ClassPath.

          Show
          Ashwin Jayaprakash added a comment - You can un-JAR the file to a directory, paste your compiled Classes into that folder and use this "exploded" directory with the original classes and your changes in the Classpath instead of having the JAR in the ClassPath.
          Hide
          Dyre Tjeldvoll added a comment -

          This issue seems to use either 'existing application impact' or 'release note needed' incorrectly, according to the description at

          http://db.apache.org/derby/DerbyBugGuidelines.html#Set+appropriate+special+%22flags%22

          I plan to remove the flags shortly unless there are additional comments.

          Show
          Dyre Tjeldvoll added a comment - This issue seems to use either 'existing application impact' or 'release note needed' incorrectly, according to the description at http://db.apache.org/derby/DerbyBugGuidelines.html#Set+appropriate+special+%22flags%22 I plan to remove the flags shortly unless there are additional comments.
          Hide
          Kristian Waagan added a comment -

          I had a quick look at this again, looking at how an in-memory backend could be wired into Derby.
          From what I can see, the easiest way is to add a single entry to BaseMonitor (and the new constant for the subSubProtocol):
          storageFactories.put( PersistentService.VFMEM, "org.apache.derby.impl.io.VFMemoryStorageFactory");

          If VFMemoryStorageFactory (or whatever the class will be called) behaves in the expected way, everything should "just work". The solution won't be optimal. The primary goal is to offer a configuration that doesn't leave any trace in the form of files and directories. Performance may be somewhat better, but the memory usage will be high. Not only will all the data be stored in memory, parts of it will be stored in multiple copies in memory.
          Note that even though the configuration will implement PersistentService, data won't be persisted beyond the life-time of the JVM running Derby. If this causes any problems with rest of the code, remains to be seen.

          A new service could be implemented as well, but I think this will require a lot more effort (feel free to comment on this if you have more information).

          I'll play some more with the idea and post my findings.

          Show
          Kristian Waagan added a comment - I had a quick look at this again, looking at how an in-memory backend could be wired into Derby. From what I can see, the easiest way is to add a single entry to BaseMonitor (and the new constant for the subSubProtocol): storageFactories.put( PersistentService.VFMEM, "org.apache.derby.impl.io.VFMemoryStorageFactory"); If VFMemoryStorageFactory (or whatever the class will be called) behaves in the expected way, everything should "just work". The solution won't be optimal. The primary goal is to offer a configuration that doesn't leave any trace in the form of files and directories. Performance may be somewhat better, but the memory usage will be high. Not only will all the data be stored in memory, parts of it will be stored in multiple copies in memory. Note that even though the configuration will implement PersistentService, data won't be persisted beyond the life-time of the JVM running Derby. If this causes any problems with rest of the code, remains to be seen. A new service could be implemented as well, but I think this will require a lot more effort (feel free to comment on this if you have more information). I'll play some more with the idea and post my findings.
          Hide
          Cheng Che Chen added a comment -

          Hi folks,

          I've implemented the interfaces org.apache.derby.io.

          {StorageFile,StorageRandomAccessFile,WritableStorageFactory}

          such that all file system operations are emulated entirely in memory, producing no file or directory on disk. I'm more than happy to contribute the implementation.

          However, I have a build problem for which I hope to get some help. To start, for simplicity, I added my source files to "trunk/java/engine/org/apache/derby/impl/io" – so that I didn't have to create/modify any "build.xml" files. I then tried to build the sources and the JAR files. The sources built with no problem – all of my classes appeared in "trunk/classes/org/apache/derby/impl/io". However, for some reason, the JAR build process did not put my classes into "derby.jar". As a result, I had to set the JAVA class path to "trunk/classes" during testing.

          I'd appreciate any help resolving this build problem.

          Thanks.

          Show
          Cheng Che Chen added a comment - Hi folks, I've implemented the interfaces org.apache.derby.io. {StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk. I'm more than happy to contribute the implementation. However, I have a build problem for which I hope to get some help. To start, for simplicity, I added my source files to "trunk/java/engine/org/apache/derby/impl/io" – so that I didn't have to create/modify any "build.xml" files. I then tried to build the sources and the JAR files. The sources built with no problem – all of my classes appeared in "trunk/classes/org/apache/derby/impl/io". However, for some reason, the JAR build process did not put my classes into "derby.jar". As a result, I had to set the JAVA class path to "trunk/classes" during testing. I'd appreciate any help resolving this build problem. Thanks.
          Hide
          Kristian Waagan added a comment -

          If your classes aren't referenced by any other classes used by Derby, they won't be included in the jar file(s).
          Assuming you are setting properties to wire in your implementation, you probably need to add your "root class(es)" to 'tools/jar/extraDBMSclasses.properties'.
          As you can see, there are many other classes in there. These are used as roots to determine which classes to include in the jars.

          Hope this helps,

          Show
          Kristian Waagan added a comment - If your classes aren't referenced by any other classes used by Derby, they won't be included in the jar file(s). Assuming you are setting properties to wire in your implementation, you probably need to add your "root class(es)" to 'tools/jar/extraDBMSclasses.properties'. As you can see, there are many other classes in there. These are used as roots to determine which classes to include in the jars. Hope this helps,
          Hide
          Cheng Che Chen added a comment - - edited

          Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.

          At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.

          I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).

          I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.

          All source files for this implementation are contained in a new sub-directory "java/engine/org/apache/derby/impl/io/memory". There is no change to any public API nor to any existing JAVA source file. One line was added to "java/engine/org/apache/derby/impl/build.xml", and one line was added to "tools/jar/extraDBMSclasses.properties", for the new source files to build.

          If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plugged things into the Derby test framework.

          Show
          Cheng Che Chen added a comment - - edited Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes. At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework. I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial ( http://db.apache.org/derby/papers/DerbyTut/index.html ). I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works. All source files for this implementation are contained in a new sub-directory "java/engine/org/apache/derby/impl/io/memory". There is no change to any public API nor to any existing JAVA source file. One line was added to "java/engine/org/apache/derby/impl/build.xml", and one line was added to "tools/jar/extraDBMSclasses.properties", for the new source files to build. If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plugged things into the Derby test framework.
          Hide
          Cheng Che Chen added a comment -

          BTW, there appears to be a discrepancy between code in the constructor of (StorageFactoryService) and documentation for StorageFactory.newStorageFile(String path).

          In the constructor of (StorageFactoryService), there is this block of code:

          rootStorageFactory = getStorageFactoryInstance( true, null, null, null);
          if( home != null)

          { StorageFile rootDir = rootStorageFactory.newStorageFile( null); rootDir.mkdirs(); }

          Now, according to the javadoc for StorageFactory.newStorageFile(String path), when the parameter (path) is null, we should return the database directory. But in the above code block, the parameter representing the database directory passed into the initialization of the storage factory, is null.

          According to the documentation for StorageFactory.init(), when the database name parameter is null, we will only work with the home directory. So I believe the author of the above code block intended (rootDir) to represent the home directory in this case (i.e., when the database parameter is null). This behavior, however, is not specified in the documentation for StorageFactory.newStorageFile(String path).

          I noticed this discrepancy during manual testing because I was getting (NullPointerException) at the call to rootDir.mkdirs() with my storage factory implementation. I worked around the problem by having my implementation of newStorageFile(String path) return the home directory when the internal database parameter is null, in the case when the input (path) parameter is null.

          In any event, there appears to be a discrepancy between code and documentation, unless I have misinterpreted the code and/or the documentation.

          Show
          Cheng Che Chen added a comment - BTW, there appears to be a discrepancy between code in the constructor of (StorageFactoryService) and documentation for StorageFactory.newStorageFile(String path). In the constructor of (StorageFactoryService), there is this block of code: rootStorageFactory = getStorageFactoryInstance( true, null, null, null); if( home != null) { StorageFile rootDir = rootStorageFactory.newStorageFile( null); rootDir.mkdirs(); } Now, according to the javadoc for StorageFactory.newStorageFile(String path), when the parameter (path) is null, we should return the database directory. But in the above code block, the parameter representing the database directory passed into the initialization of the storage factory, is null. According to the documentation for StorageFactory.init(), when the database name parameter is null, we will only work with the home directory. So I believe the author of the above code block intended (rootDir) to represent the home directory in this case (i.e., when the database parameter is null). This behavior, however, is not specified in the documentation for StorageFactory.newStorageFile(String path). I noticed this discrepancy during manual testing because I was getting (NullPointerException) at the call to rootDir.mkdirs() with my storage factory implementation. I worked around the problem by having my implementation of newStorageFile(String path) return the home directory when the internal database parameter is null, in the case when the input (path) parameter is null. In any event, there appears to be a discrepancy between code and documentation, unless I have misinterpreted the code and/or the documentation.
          Hide
          Kristian Waagan added a comment -

          Cheng,

          As it happens, I have a working prototype as well. It closely follows the approach taken by Stephen Fitch in the original patch.
          If possible, I think we should somehow try to coordinate our work. I'd like to spend a few more days cleaning up the initial code contribution, as it is pretty fresh.

          Regarding testing, the easiest way to do it is to override the storage factory for PersistentService.DIRECTORY in BaseMonitor. Then you can just run the tests as normal, either by using JUnit directly or by running ant junitreport.
          Currently, I'm getting 24 failures and 66 errors when running junitreport. Somes of these are to be expected, as some tests explicitly tries to access files in the database directory. I also note problems in general with encryption, backup and restoreFrom (in the url) etc, but I suspect these tests require the database files to exist as well.
          Also, some tests will just fail because all the data is stored in memory, for instance 'ant junit-lowmem'.
          Lastly, the test framework has a decorator to delete a database and/or a directory. Since it is working against the file system only, it doesn't manage to delete an in-memory database.

          Note that bugs in the prototype tend to hide if they are triggered during boot. If things go wrong, I recommend to check derby.log before starting debugging.

          I'll post my code (and the test results) in a day or two, and we should consider how to best cooperate on this issue.

          Regards,

          Show
          Kristian Waagan added a comment - Cheng, As it happens, I have a working prototype as well. It closely follows the approach taken by Stephen Fitch in the original patch. If possible, I think we should somehow try to coordinate our work. I'd like to spend a few more days cleaning up the initial code contribution, as it is pretty fresh. Regarding testing, the easiest way to do it is to override the storage factory for PersistentService.DIRECTORY in BaseMonitor. Then you can just run the tests as normal, either by using JUnit directly or by running ant junitreport. Currently, I'm getting 24 failures and 66 errors when running junitreport. Somes of these are to be expected, as some tests explicitly tries to access files in the database directory. I also note problems in general with encryption, backup and restoreFrom (in the url) etc, but I suspect these tests require the database files to exist as well. Also, some tests will just fail because all the data is stored in memory, for instance 'ant junit-lowmem'. Lastly, the test framework has a decorator to delete a database and/or a directory. Since it is working against the file system only, it doesn't manage to delete an in-memory database. Note that bugs in the prototype tend to hide if they are triggered during boot. If things go wrong, I recommend to check derby.log before starting debugging. I'll post my code (and the test results) in a day or two, and we should consider how to best cooperate on this issue. Regards,
          Hide
          Cheng Che Chen added a comment -

          Sometime back I played a bit with the original solution posted in this thread. I remember that it produced some files and directories on disk although it was supposed to be an in-memory storage factory. But if it works and suits the tasks at hand, we should let it be.

          Show
          Cheng Che Chen added a comment - Sometime back I played a bit with the original solution posted in this thread. I remember that it produced some files and directories on disk although it was supposed to be an in-memory storage factory. But if it works and suits the tasks at hand, we should let it be.
          Hide
          Kristian Waagan added a comment -

          Yes, me too. However, since I was interested in learning a bit more about Derby and because I didn't know the state of the existing patch, I started from scratch.
          It is my impression that the existing patch isn't "production ready".

          I think it would be cool to compare our approaches. Maybe we can draw from both of them and create an even better solution!
          The only thing my prototype leaves on disk, is derby.log. I think this can be easily changed, but often you would want to consult the log anyway.
          The goal of my implementation is to avoid creating files/dirs on disk, which for some people is attractive for running tests. I believe there are also performance benefits, especially for update operations, at the cost of increased memory usage of course. The database creation time should also be shorter, but I haven't obtained any numbers on this yet.

          Show
          Kristian Waagan added a comment - Yes, me too. However, since I was interested in learning a bit more about Derby and because I didn't know the state of the existing patch, I started from scratch. It is my impression that the existing patch isn't "production ready". I think it would be cool to compare our approaches. Maybe we can draw from both of them and create an even better solution! The only thing my prototype leaves on disk, is derby.log. I think this can be easily changed, but often you would want to consult the log anyway. The goal of my implementation is to avoid creating files/dirs on disk, which for some people is attractive for running tests. I believe there are also performance benefits, especially for update operations, at the cost of increased memory usage of course. The database creation time should also be shorter, but I haven't obtained any numbers on this yet.
          Hide
          David Sims added a comment -

          I'm following this thread with great interest. Many thanks to you guys for working on this.

          > The only thing my prototype leaves on disk, is derby.log. I think this can be easily
          > changed, but often you would want to consult the log anyway.

          For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit.

          Thank you for your work,
          David

          Show
          David Sims added a comment - I'm following this thread with great interest. Many thanks to you guys for working on this. > The only thing my prototype leaves on disk, is derby.log. I think this can be easily > changed, but often you would want to consult the log anyway. For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit. Thank you for your work, David
          Hide
          Kristian Waagan added a comment -

          Thank you for the feedback, David.

          > For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit.

          What do you want to happen with derby.log, which may contain critical information about errors happening in Derby?
          Note that the file by default will be overwritten when you restart Derby, so it shouldn't eat up your disk space. Since very little is written to it under normal operation, it shouldn't degrade performance. If you fire up lots of instances in different directories (without setting derby.system.home), you do get 'derby.log' files lying around though.

          If you want to discard the information, I think you have the possibility to do that yourself using one of the derby.stream.error properties. To make it even easier, we could add a stream class to the Derby engine jar file that simply discards the information (if specifying null causes the information to be printed to std err).

          My personal choice would be to not discard error information by default though, and I think it should be a conscientious choice made by the user / developer, but feel free to disagree

          Show
          Kristian Waagan added a comment - Thank you for the feedback, David. > For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit. What do you want to happen with derby.log, which may contain critical information about errors happening in Derby? Note that the file by default will be overwritten when you restart Derby, so it shouldn't eat up your disk space. Since very little is written to it under normal operation, it shouldn't degrade performance. If you fire up lots of instances in different directories (without setting derby.system.home), you do get 'derby.log' files lying around though. If you want to discard the information, I think you have the possibility to do that yourself using one of the derby.stream.error properties. To make it even easier, we could add a stream class to the Derby engine jar file that simply discards the information (if specifying null causes the information to be printed to std err). My personal choice would be to not discard error information by default though, and I think it should be a conscientious choice made by the user / developer, but feel free to disagree
          Hide
          David Sims added a comment -

          Thanks, Kristian.

          In my situation, I would use Derby as a strictly embedded database, and I don't want my users to even be confused by the presence of a derby.log file, however empty it may be. The experience with my users tells me that it's best just to have absolutely nothing. I can deal with the usual SQLException errors that come along.

          However you and the others might feel that should best be configured in Derby, I leave up to you all. I would just like to chime in and request that there be a way to configure things so that absolutely nothing is on disk, including derby.log – however that might work out.

          Thanks,
          David

          Show
          David Sims added a comment - Thanks, Kristian. In my situation, I would use Derby as a strictly embedded database, and I don't want my users to even be confused by the presence of a derby.log file, however empty it may be. The experience with my users tells me that it's best just to have absolutely nothing. I can deal with the usual SQLException errors that come along. However you and the others might feel that should best be configured in Derby, I leave up to you all. I would just like to chime in and request that there be a way to configure things so that absolutely nothing is on disk, including derby.log – however that might work out. Thanks, David
          Hide
          Cheng Che Chen added a comment -

          I believe "derby.log" is Derby's execution log, and operations on that log would probably not be going through the database storage layer. A run-time option to disable execution logging should be a separate work item (please file a ticket) and not be included in the resolution of this issue, in my opinion.

          Show
          Cheng Che Chen added a comment - I believe "derby.log" is Derby's execution log, and operations on that log would probably not be going through the database storage layer. A run-time option to disable execution logging should be a separate work item (please file a ticket) and not be included in the resolution of this issue, in my opinion.
          Hide
          Kathey Marsden added a comment -

          As Kristian mentioned, you can disable error logging by using derby.stream.error.field or derby.stream.error.method. See
          http://db.apache.org/derby/docs/10.4/tuning/rtunproper35028.html
          http://db.apache.org/derby/docs/10.4/tuning/rtunproper33027.html

          So we probably don't need to file a separate issue for that.

          Show
          Kathey Marsden added a comment - As Kristian mentioned, you can disable error logging by using derby.stream.error.field or derby.stream.error.method. See http://db.apache.org/derby/docs/10.4/tuning/rtunproper35028.html http://db.apache.org/derby/docs/10.4/tuning/rtunproper33027.html So we probably don't need to file a separate issue for that.
          Hide
          Kristian Waagan added a comment -

          The patch 'derby-646-2a-vfmem_first_rev.diff' is the first revision of an in-memory backend for Derby.

          The implementation takes the same general approach as the original patch contributed by Stephen Fitch, but it is implemented from scratch.
          This was a choice I made because the state of the original patch was unknown, and I wanted to learn more about the internals of Derby in this area.

          Since this is the first revision, there are limitations;
          o The maximum "file size" is 256 MB, 512 MB or 2 GB depending on the page size used.
          The system tries to make the blocks used to store data the same size as the pages used by Derby. When the page size is "indeterminable", 4 KB is used by default.
          If you hit the 256 MB limit, you have two options; set the page size to 32 KB, or modify BlockedByteArray.
          I think that either the max size should be configurable via properties, or the BlockedByteArray should be extended with a better growth functionality. This is basically a tradeoff between memory usage for smaller files, and CPU usage (copying bytes). As an alternative, I think one can avoid both excessive memory overhead and byte copying by adding complexity
          o The in-memory backend is not enabled for the client driver.
          I haven't investigated this, but if you want to test with the client driver, replace the storage factory class for PersistentStorage.DIRECTORY in BaseMonitor with the VFMemoryStoreageFactory.
          This will make the in-memory backend the default backend, and none of you databases will be persisted.
          I think some handling code in the client driver has to be added to make the in-memory backend available there.
          o There is no easy way to delete a database, except for restarting the JVM.
          If this is a problem for your usage, there is a purgeDatabase-method in the storage factory (a database is currently named by an absolute file path in the storage factory).

          [How to test]
          1) Download the patch (2a), apply it and build Derby.
          2) Connect to a database using the following prefix in the JDBC connection url: "jdbc:derby:mem" (remember to create the database the first time you connect to it).

          WARNING: I have only tested this on OpenSolaris. It would be nice if someone running Windows could take the patch for a test-spin.
          I'm sure there are more limitations and problems, but I'd like to get this out in the community as soon as possible. I have managed to run most of the existing regression tests with the patch, so it should be usable.

          Feedback is welcome.

          Show
          Kristian Waagan added a comment - The patch 'derby-646-2a-vfmem_first_rev.diff' is the first revision of an in-memory backend for Derby. The implementation takes the same general approach as the original patch contributed by Stephen Fitch, but it is implemented from scratch. This was a choice I made because the state of the original patch was unknown, and I wanted to learn more about the internals of Derby in this area. Since this is the first revision, there are limitations; o The maximum "file size" is 256 MB, 512 MB or 2 GB depending on the page size used. The system tries to make the blocks used to store data the same size as the pages used by Derby. When the page size is "indeterminable", 4 KB is used by default. If you hit the 256 MB limit, you have two options; set the page size to 32 KB, or modify BlockedByteArray. I think that either the max size should be configurable via properties, or the BlockedByteArray should be extended with a better growth functionality. This is basically a tradeoff between memory usage for smaller files, and CPU usage (copying bytes). As an alternative, I think one can avoid both excessive memory overhead and byte copying by adding complexity o The in-memory backend is not enabled for the client driver. I haven't investigated this, but if you want to test with the client driver, replace the storage factory class for PersistentStorage.DIRECTORY in BaseMonitor with the VFMemoryStoreageFactory. This will make the in-memory backend the default backend, and none of you databases will be persisted. I think some handling code in the client driver has to be added to make the in-memory backend available there. o There is no easy way to delete a database, except for restarting the JVM. If this is a problem for your usage, there is a purgeDatabase-method in the storage factory (a database is currently named by an absolute file path in the storage factory). [How to test] 1) Download the patch (2a), apply it and build Derby. 2) Connect to a database using the following prefix in the JDBC connection url: "jdbc:derby:mem" (remember to create the database the first time you connect to it). WARNING: I have only tested this on OpenSolaris. It would be nice if someone running Windows could take the patch for a test-spin. I'm sure there are more limitations and problems, but I'd like to get this out in the community as soon as possible. I have managed to run most of the existing regression tests with the patch, so it should be usable. Feedback is welcome.
          Hide
          Kristian Waagan added a comment -

          Forgot to say, but if you want to spend time on some of the known weaker spots of the patch, search the diff for "TODO".

          Also, I'm wondering if it would be wise or not to ditch the self-made path handling and rely on java.io.File instead.

          Show
          Kristian Waagan added a comment - Forgot to say, but if you want to spend time on some of the known weaker spots of the patch, search the diff for "TODO". Also, I'm wondering if it would be wise or not to ditch the self-made path handling and rely on java.io.File instead.
          Hide
          Cheng Che Chen added a comment - - edited

          Hi folks,

          Attached (derby-646-20090222.stat, derby-646-20090222.diff) is my implementation of the interfaces org.apache.derby.io.

          {StorageFile,StorageRandomAccessFile,WritableStorageFactory}

          such that all file system operations are emulated entirely in memory, producing no file or directory on disk.

          A brief description of the content:

          • java\engine\org\apache\derby\impl\io\memory

          This directory contains the source files for the implementation of the database storage factory interfaces.

          • java\testing\org\apache\derbyTesting\functionTests\tests\memoryIO

          This directory contains the source files for the JUnit tests. It consists of three sets of fairly extensive unit tests of the basic building blocks of my implementation. They started out as five sets of stand-alone JUnit tests and were converted to conform to Derby's JUnit test framework. In the process of conversion, two sets of stand-alone tests were dropped because they use on-disk operations to verify the correctness of in-memory operations, and Derby's test framework does not allow arbitrary operations on disk, as far as I can tell. These unit tests run only in the embedded mode of Derby's test framework, because they make no DB connections.

          In addition, there is one functional test consisting of a simple but non-trivial clustering application. It is non-trivial because clustering is an important application in many engineering and scientific disciplines; it is simple in that it computes clustering of numerical data in one dimensional space – the simplest case. The application implements the same algorithm in both JAVA (computed by the JUnit test case object itself) and SQL (computed by Derby using the in-memory storage factory) and cross-checks the computed results, within some floating-point tolerance. The SQL portion of the application computes quite a few CREATE, INSERT, DELETE, SELECT, JOIN, GROUP BY, ORDER BY, DROP; and so should exercise the storage layer to a good extent. This functional test runs in both the embedded mode and the client mode of Derby's test framework.

          The sub-sub-protocol associated with this in-memory storage factory is named "memory". So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".


          HELP!!! I've gotten the tests compiled properly with "ant all". The compiled class files show up under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". But again I'm running into the problem of not knowing how to get them included in the "derbyTesting.jar" file.

          Nor did I figure out how to have them automatically executed with "ant junit-all", "ant junitreport", etc. For now, this is what I do after "ant all" to execute the tests on Windows:

          java -cp .\tools\java\junit.jar;.\classes junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.memoryIO._Suite

          (BTW, when I execute the above command, the disk activity LED on my computer goes wild during the functional test. That doesn't happen with the stand-alone version of the same functional test (before conversion to run under Derby's test framework). Perhaps I have mis-configured something in the test object itself such that the above command does not actually test the in-memory storage factory?)


          Existing files modified by this patch:

          • java\engine\org\apache\derby\impl\build.xml
            tools\jar\extraDBMSclasses.properties

          These two files were modified so that my implementation would be compiled and included in Derby's JAR files.

          • java\engine\org\apache\derby\iapi\services\monitor\PersistentService.java
            java\engine\org\apache\derby\impl\services\monitor\BaseMonitor.java

          These two files were modified only because I did not feel like typing "-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.memory.MemoryStorageFactory" for every run. I simply added a mapping from the "memory" service to the "org.apache.derby.impl.io.memory.MemoryStorageFactory" class in BaseMonitor.

          • java\testing\build.xml

          This file was modified so that my tests would be compiled into class files under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO".


          Thanks.

          Show
          Cheng Che Chen added a comment - - edited Hi folks, Attached (derby-646-20090222.stat, derby-646-20090222.diff) is my implementation of the interfaces org.apache.derby.io. {StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk. A brief description of the content: java\engine\org\apache\derby\impl\io\memory This directory contains the source files for the implementation of the database storage factory interfaces. java\testing\org\apache\derbyTesting\functionTests\tests\memoryIO This directory contains the source files for the JUnit tests. It consists of three sets of fairly extensive unit tests of the basic building blocks of my implementation. They started out as five sets of stand-alone JUnit tests and were converted to conform to Derby's JUnit test framework. In the process of conversion, two sets of stand-alone tests were dropped because they use on-disk operations to verify the correctness of in-memory operations, and Derby's test framework does not allow arbitrary operations on disk, as far as I can tell. These unit tests run only in the embedded mode of Derby's test framework, because they make no DB connections. In addition, there is one functional test consisting of a simple but non-trivial clustering application. It is non-trivial because clustering is an important application in many engineering and scientific disciplines; it is simple in that it computes clustering of numerical data in one dimensional space – the simplest case. The application implements the same algorithm in both JAVA (computed by the JUnit test case object itself) and SQL (computed by Derby using the in-memory storage factory) and cross-checks the computed results, within some floating-point tolerance. The SQL portion of the application computes quite a few CREATE, INSERT, DELETE, SELECT, JOIN, GROUP BY, ORDER BY, DROP; and so should exercise the storage layer to a good extent. This functional test runs in both the embedded mode and the client mode of Derby's test framework. The sub-sub-protocol associated with this in-memory storage factory is named "memory". So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb". HELP!!! I've gotten the tests compiled properly with "ant all". The compiled class files show up under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". But again I'm running into the problem of not knowing how to get them included in the "derbyTesting.jar" file. Nor did I figure out how to have them automatically executed with "ant junit-all", "ant junitreport", etc. For now, this is what I do after "ant all" to execute the tests on Windows: java -cp .\tools\java\junit.jar;.\classes junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.memoryIO._Suite (BTW, when I execute the above command, the disk activity LED on my computer goes wild during the functional test. That doesn't happen with the stand-alone version of the same functional test (before conversion to run under Derby's test framework). Perhaps I have mis-configured something in the test object itself such that the above command does not actually test the in-memory storage factory?) Existing files modified by this patch: java\engine\org\apache\derby\impl\build.xml tools\jar\extraDBMSclasses.properties These two files were modified so that my implementation would be compiled and included in Derby's JAR files. java\engine\org\apache\derby\iapi\services\monitor\PersistentService.java java\engine\org\apache\derby\impl\services\monitor\BaseMonitor.java These two files were modified only because I did not feel like typing "-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.memory.MemoryStorageFactory" for every run. I simply added a mapping from the "memory" service to the "org.apache.derby.impl.io.memory.MemoryStorageFactory" class in BaseMonitor. java\testing\build.xml This file was modified so that my tests would be compiled into class files under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". Thanks.
          Hide
          Kristian Waagan added a comment -

          We now have two candidate solutions. How do we proceed?

          I would very much like to get an in-memory back end included in the upcoming
          10.5 release. In my opinion this should be pretty safe, as almost no existing
          code is modified and the user has to take an active step to use the in-memory
          back end.

          To start mapping the performance characteristics of the two solutions, I ran
          some of the performance tests we have. Note that these are based on only a few
          runs, and a run time of 60 seconds only (pluss 30 seconds warmup). The numbers
          are average tps (transactions per second).
          There is some variation in the tests, so where the numbers differ only by a very
          small percentage, I think we can assume the performance is comparable for now.

          Hardware #1: 32 hardware execution threads, 1200 MHz.
          sr_update sr_select bank_tx
          1 thread
          memory 1963 3059 823
          vfmem 2307 3753 878
          8 threads
          memory 7662 14321 2163
          vfmem 12692 17695 2294
          32 threads
          memory 7357 13984 2076
          vfmem 12219 16546 2144

          Hardware #2: Dual core Opteron, 2.4 GHz
          sr_update sr_select bank_tx
          1 thread
          memory 12257 20295 5123
          vfmem 12288 20657 5189
          4 threads
          memory 15064 32117 6120
          vfmem 15857 32629 6115
          8 threads
          memory 16682 31257 5767
          vfmem 16357 31361 5775

          Hardware #3: 2xDual core Opteron, 2.8 GHz
          sr_update sr_select bank_tx
          1 thread
          memory 13548 22184 5805
          vfmem 13705 21770 5765
          4 threads
          memory 29019 57861 10443
          vfmem 29727 58053 10479
          8 threads
          memory 26348 46580 9372
          vfmem 26676 46805 9347
          16 threads
          memory 24500 44249 8976
          vfmem 24883 43832 9014

          Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page cache size
          40 1000 15000
          1 thread
          memory 9932 12234 19207
          vfmem 10075 12223 18995
          8 threads
          memory 12898 16540 26393
          vfmem 13148 16662 28017

          Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page size
          Page cache size | 40 | | 1000 |
          4 KB 32 KB 4 KB 32 KB
          1 thread
          memory 9902 2902 12369 20168
          vfmem 10342 3247 12059 19801
          8 threads
          memory 12926 3599 16523 27333
          vfmem 13261 4059 16592 28361

          As far as I can see from these simple performance tests, the only configuration where the two solutions differs significantly, is when running on a CMT (chip multi threading) machine. Of the loads I have tested, this is most clearly visible with update load.
          I have not investigated this further.

          Another thing one could look into, is the memory usage of the two solutions.

          Show
          Kristian Waagan added a comment - We now have two candidate solutions. How do we proceed? I would very much like to get an in-memory back end included in the upcoming 10.5 release. In my opinion this should be pretty safe, as almost no existing code is modified and the user has to take an active step to use the in-memory back end. To start mapping the performance characteristics of the two solutions, I ran some of the performance tests we have. Note that these are based on only a few runs, and a run time of 60 seconds only (pluss 30 seconds warmup). The numbers are average tps (transactions per second). There is some variation in the tests, so where the numbers differ only by a very small percentage, I think we can assume the performance is comparable for now. Hardware #1: 32 hardware execution threads, 1200 MHz. sr_update sr_select bank_tx 1 thread memory 1963 3059 823 vfmem 2307 3753 878 8 threads memory 7662 14321 2163 vfmem 12692 17695 2294 32 threads memory 7357 13984 2076 vfmem 12219 16546 2144 Hardware #2: Dual core Opteron, 2.4 GHz sr_update sr_select bank_tx 1 thread memory 12257 20295 5123 vfmem 12288 20657 5189 4 threads memory 15064 32117 6120 vfmem 15857 32629 6115 8 threads memory 16682 31257 5767 vfmem 16357 31361 5775 Hardware #3: 2xDual core Opteron, 2.8 GHz sr_update sr_select bank_tx 1 thread memory 13548 22184 5805 vfmem 13705 21770 5765 4 threads memory 29019 57861 10443 vfmem 29727 58053 10479 8 threads memory 26348 46580 9372 vfmem 26676 46805 9347 16 threads memory 24500 44249 8976 vfmem 24883 43832 9014 Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page cache size 40 1000 15000 1 thread memory 9932 12234 19207 vfmem 10075 12223 18995 8 threads memory 12898 16540 26393 vfmem 13148 16662 28017 Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page size Page cache size | 40 | | 1000 | 4 KB 32 KB 4 KB 32 KB 1 thread memory 9902 2902 12369 20168 vfmem 10342 3247 12059 19801 8 threads memory 12926 3599 16523 27333 vfmem 13261 4059 16592 28361 As far as I can see from these simple performance tests, the only configuration where the two solutions differs significantly, is when running on a CMT (chip multi threading) machine. Of the loads I have tested, this is most clearly visible with update load. I have not investigated this further. Another thing one could look into, is the memory usage of the two solutions.
          Hide
          Kristian Waagan added a comment -

          'derby-646-performance_comparison_1a.txt' is my previous post as a text file. The formatting here is a little better...

          Cheng wrote:


          So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".


          Regarding the client/server URL, should we change the client driver to understand the following URL instead?
          "jdbc:derby:memory://localhost:1527/MyTestDb"

          It looks cleaner to me.

          Show
          Kristian Waagan added a comment - 'derby-646-performance_comparison_1a.txt' is my previous post as a text file. The formatting here is a little better... Cheng wrote: So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb". Regarding the client/server URL, should we change the client driver to understand the following URL instead? "jdbc:derby:memory://localhost:1527/MyTestDb" It looks cleaner to me.
          Hide
          Kathey Marsden added a comment -

          Kristian asked:
          >We now have two candidate solutions. How do we proceed?

          I haven't looked at the code but I guess the first thing we need to consider is legal. Is Cheng Che Chen willing to file an ICLA?

          Show
          Kathey Marsden added a comment - Kristian asked: >We now have two candidate solutions. How do we proceed? I haven't looked at the code but I guess the first thing we need to consider is legal. Is Cheng Che Chen willing to file an ICLA?
          Hide
          Mike Matrigali added a comment -

          As to how to proceed, first is there an icla for cheng's contribution?

          Is it possible for either of these patches to run through the entire derby test suite, in in-memory mode?

          how does either compare to "test mode" with the page cache set as big as expected in-memory
          use?

          Kristian, or any other committer, can you take a look at both implementations and comment
          on any
          strength's/weeknesses from both. As you suggest it would be nice if we could get a combined
          agreed implementation from both.

          If goal is really the fast approaching 10.5, then it would be nice to have some sort of upper level user documentation. How to use it, what are limits, is all functionality supported. What happens in case it runs out of memory, or internal limits.

          Show
          Mike Matrigali added a comment - As to how to proceed, first is there an icla for cheng's contribution? Is it possible for either of these patches to run through the entire derby test suite, in in-memory mode? how does either compare to "test mode" with the page cache set as big as expected in-memory use? Kristian, or any other committer, can you take a look at both implementations and comment on any strength's/weeknesses from both. As you suggest it would be nice if we could get a combined agreed implementation from both. If goal is really the fast approaching 10.5, then it would be nice to have some sort of upper level user documentation. How to use it, what are limits, is all functionality supported. What happens in case it runs out of memory, or internal limits.
          Hide
          Cheng Che Chen added a comment -

          I believe I have included the Apache license header in all the files and had checked the box granting rights to Apache when I submitted those files. According to the comment by Andrew McIntyre on 16/Mar/07 10:32 AM in this thread, that should be sufficient, no? To have an ICLA on file, I'll have to fill it out and mail it to Apache – this will take some time. Let me know if this is truly required. Thanks.

          Show
          Cheng Che Chen added a comment - I believe I have included the Apache license header in all the files and had checked the box granting rights to Apache when I submitted those files. According to the comment by Andrew McIntyre on 16/Mar/07 10:32 AM in this thread, that should be sufficient, no? To have an ICLA on file, I'll have to fill it out and mail it to Apache – this will take some time. Let me know if this is truly required. Thanks.
          Hide
          V.Narayanan added a comment -

          can u please assign the issue to your name? Also for major contributions a icla is generally required and this looks like a major feature. my two paisa!

          Show
          V.Narayanan added a comment - can u please assign the issue to your name? Also for major contributions a icla is generally required and this looks like a major feature. my two paisa!
          Hide
          Kristian Waagan added a comment -

          Here are some results from running 'ant junitreport' with patch 2a
          (vfmem).

          The prefix 'org.apache.derbyTesting.' has been removed for brevity.
          The first number is the number of tests run with the vfmem back end,
          the second number is the number of tests run with a clean trunk.
          Note that I changes the memory back end to be the default back end in
          this run (modified the entry for DIRECTORY storage factory in BaseMonitor), and
          I also changed DropDatabaseSetup to delete the in-memory database.

          Suites run without failures:
          junit.EnvTest (9/9)
          functionTests.tests.jdbcapi.AutoloadTest (3/3)
          functionTests.tests.jdbcapi.JDBCDriversEmbeddedTest (8/8)
          functionTests.tests.jdbcapi.JDBCDriversClientTest (4/4)
          functionTests.tests.jdbcapi.JDBCDriversAllTest (12/12)
          functionTests.tests.tools._Suite (87/87)
          functionTests.tests.demo._Suite (2/2)
          functionTests.tests.engine._Suite (11/11)
          functionTests.suites.EncryptionSuite (7/7)
          functionTests.tests.jdbc4._Suite (896/898) // One test added later
          functionTests.tests.management._Suite (100/100)

          The follwing tests had failures and/or errors:
          functionTests.tests.derbynet._Suite (t=122/f=1/e=0)
          (f) ShutDownDBWhenNSShutsDownTest.testDatabasesShutDownWhenNSShutdown
          functionTests.tests.lang._Suite (t=1473/f=4/e=2)
          (f) DatabaseClassLoadingTest.testCreateDatabaseJar
          (f) DatabaseClassLoadingTest.testDatabaseInJar
          (f) ConnectTest.testDBName
          (f) ConnectTest.testDBName
          (e) DatabaseClassLoadingTest.testHackedJarReplacedClass
          (e) DBInJarTest.testConnectDBInJar
          functionTests.tests.jdbcapi._Suite (t=6301/f=2/e=0)
          (f) PoolXADSCreateShutdownDBTest.testXADS
          (f) PoolXADSCreateShutdownDBTest.testXADS
          functionTests.tests.store._Suite (t=98/f=4/e=19)
          (f) BootAllTest.testSettingBootAllPropertyWithHomePropertySet
          (f) OSReadOnlyTest.testOSReadOnly
          (f) OSReadOnlyTest.testOSReadOnly
          (f) BackupRestoreTest.testDerby3875
          o Ok, test is trying to correct file on disk.
          (e) OfflineBackupTest.testCreateFromRestoreFrom
          6x (e) EncryptionKeyAESTest
          6x (e) EncryptionKeyBlowfishTest
          6x (e) EncryptionKeyDESTest
          unitTests.junit._Suite
          (crashes due to ant/junit problem, will rerun later)
          (e) AccessControlException
          functionTests.tests.upgradeTests._Suite (t=420/f=12/e=42)
          (needs investigation)
          functionTests.tests.memory._Suite
          o Ok, fails because it sets the heap size really low and tries
          to insert data that is bigger than the heap.

          To sum up, I'm currently getting 23 failures and 65 errors.
          The time it takes to run the suites is around 45 minutes.
          I believe the 'ant junitreport' target is roughly equal to running
          suites.All.

          Show
          Kristian Waagan added a comment - Here are some results from running 'ant junitreport' with patch 2a (vfmem). The prefix 'org.apache.derbyTesting.' has been removed for brevity. The first number is the number of tests run with the vfmem back end, the second number is the number of tests run with a clean trunk. Note that I changes the memory back end to be the default back end in this run (modified the entry for DIRECTORY storage factory in BaseMonitor), and I also changed DropDatabaseSetup to delete the in-memory database. Suites run without failures: junit.EnvTest (9/9) functionTests.tests.jdbcapi.AutoloadTest (3/3) functionTests.tests.jdbcapi.JDBCDriversEmbeddedTest (8/8) functionTests.tests.jdbcapi.JDBCDriversClientTest (4/4) functionTests.tests.jdbcapi.JDBCDriversAllTest (12/12) functionTests.tests.tools._Suite (87/87) functionTests.tests.demo._Suite (2/2) functionTests.tests.engine._Suite (11/11) functionTests.suites.EncryptionSuite (7/7) functionTests.tests.jdbc4._Suite (896/898) // One test added later functionTests.tests.management._Suite (100/100) The follwing tests had failures and/or errors: functionTests.tests.derbynet._Suite (t=122/f=1/e=0) (f) ShutDownDBWhenNSShutsDownTest.testDatabasesShutDownWhenNSShutdown functionTests.tests.lang._Suite (t=1473/f=4/e=2) (f) DatabaseClassLoadingTest.testCreateDatabaseJar (f) DatabaseClassLoadingTest.testDatabaseInJar (f) ConnectTest.testDBName (f) ConnectTest.testDBName (e) DatabaseClassLoadingTest.testHackedJarReplacedClass (e) DBInJarTest.testConnectDBInJar functionTests.tests.jdbcapi._Suite (t=6301/f=2/e=0) (f) PoolXADSCreateShutdownDBTest.testXADS (f) PoolXADSCreateShutdownDBTest.testXADS functionTests.tests.store._Suite (t=98/f=4/e=19) (f) BootAllTest.testSettingBootAllPropertyWithHomePropertySet (f) OSReadOnlyTest.testOSReadOnly (f) OSReadOnlyTest.testOSReadOnly (f) BackupRestoreTest.testDerby3875 o Ok, test is trying to correct file on disk. (e) OfflineBackupTest.testCreateFromRestoreFrom 6x (e) EncryptionKeyAESTest 6x (e) EncryptionKeyBlowfishTest 6x (e) EncryptionKeyDESTest unitTests.junit._Suite (crashes due to ant/junit problem, will rerun later) (e) AccessControlException functionTests.tests.upgradeTests._Suite (t=420/f=12/e=42) (needs investigation) functionTests.tests.memory._Suite o Ok, fails because it sets the heap size really low and tries to insert data that is bigger than the heap. To sum up, I'm currently getting 23 failures and 65 errors. The time it takes to run the suites is around 45 minutes. I believe the 'ant junitreport' target is roughly equal to running suites.All.
          Hide
          Kathey Marsden added a comment -

          If you have access to a scanner, you can email your scanned ICLA to secretary@apache.org (see http://www.apache.org/licenses/#clas).

          Show
          Kathey Marsden added a comment - If you have access to a scanner, you can email your scanned ICLA to secretary@apache.org (see http://www.apache.org/licenses/#clas ).
          Hide
          Rick Hillegas added a comment -

          For the first time in the history of the Derby community, two complete implementations of a significant feature have surfaced almost simultaneously. I'm afraid we have to pick one. I hope that the other contributor does not feel snubbed. Thanks to Kristian and Cheng for these two great submissions.

          I have taken a quick look at the patches and at the commentary on this JIRA. I prefer Kristian's implementation for the following reasons:

          1) The coding style is slightly easier for me to understand.

          2) According to the performance data which Kristian posted, his implementation is slightly faster.

          I hope that Cheng's ICLA arrives soon so that we can mine his patch for ideas that can be layered on top of Kristian's work. For instance,

          A) Cheng's solution lets you snapshot the database to disk at shutdown time. I think some users will find this useful. I understand that we could simulate this behavior using backup and restore, but I would not discount the appeal of this automatic behavior. Perhaps the api here could be an additional subprotocol.

          B) Cheng's solution has a finalizer method. This seems particularly useful for delivering (A) in the face of a system crash. But winddown logic seems generally useful for a complicated program like an rdbms.

          C) Cheng provides some additional tests. Some of these tests may be implementation-agnostic and worth running against Kristian's implementation. It would be helpful if Cheng could describe these tests in greater detail so that we can decide whether to incorporate them.

          Thanks again to both Kristian and Cheng.

          Show
          Rick Hillegas added a comment - For the first time in the history of the Derby community, two complete implementations of a significant feature have surfaced almost simultaneously. I'm afraid we have to pick one. I hope that the other contributor does not feel snubbed. Thanks to Kristian and Cheng for these two great submissions. I have taken a quick look at the patches and at the commentary on this JIRA. I prefer Kristian's implementation for the following reasons: 1) The coding style is slightly easier for me to understand. 2) According to the performance data which Kristian posted, his implementation is slightly faster. I hope that Cheng's ICLA arrives soon so that we can mine his patch for ideas that can be layered on top of Kristian's work. For instance, A) Cheng's solution lets you snapshot the database to disk at shutdown time. I think some users will find this useful. I understand that we could simulate this behavior using backup and restore, but I would not discount the appeal of this automatic behavior. Perhaps the api here could be an additional subprotocol. B) Cheng's solution has a finalizer method. This seems particularly useful for delivering (A) in the face of a system crash. But winddown logic seems generally useful for a complicated program like an rdbms. C) Cheng provides some additional tests. Some of these tests may be implementation-agnostic and worth running against Kristian's implementation. It would be helpful if Cheng could describe these tests in greater detail so that we can decide whether to incorporate them. Thanks again to both Kristian and Cheng.
          Hide
          Rick Hillegas added a comment -

          I also would like to see us deliver an in-memory implementation in 10.5. I recommend that we check in one of these implementations but treat it as experimental code, not supported by official user documentation. A writeup of how to use the feature can appear on our wiki. After we release 10.5, we can solicit user feedback and adjust the api accordingly. Then in 10.6 we can expose this feature as solid and official with supporting user documentation. I think that this feature is important enough that we should consider spinning a 10.6 release a couple months after 10.5 if our users provide fast feedback.

          Show
          Rick Hillegas added a comment - I also would like to see us deliver an in-memory implementation in 10.5. I recommend that we check in one of these implementations but treat it as experimental code, not supported by official user documentation. A writeup of how to use the feature can appear on our wiki. After we release 10.5, we can solicit user feedback and adjust the api accordingly. Then in 10.6 we can expose this feature as solid and official with supporting user documentation. I think that this feature is important enough that we should consider spinning a 10.6 release a couple months after 10.5 if our users provide fast feedback.
          Hide
          Cheng Che Chen added a comment -

          I have just emailed scanned images of signed ICLA to secretary@apache.org, as instructed by Kathey Marsden.

          Regarding Rick Hillegas' question about the tests included in my patch, there are three sets of unit tests and one functional test.

          The unit tests verify the correctness of my internal data structures. They are probably not useful to other implementations, unless, for example, they have similar properties to be verified.

          The functional test implements an application in both JAVA and SQL and cross-checks the computed results. It can indeed be applied to any implementation of storage factory by simply changing the database name for the SQL portion of the test object. The functional test consists of the following files: "GenMog.java", "SampMog.java", "ClusMog.java", "ClusMogSQL.java", and "MogTest.java" – the last file is the JUnit test object.

          However, I have some doubt about whether I have configured/used Derby's test framework properly in "MogTest.java". This file started out as a stand-alone JUnit test. When I was running it stand-alone, it would incur no disk activity. After I converted the file to run under Derby's test framework, the disk activity LED on my computer would go wild every time I ran it. I wonder if I have unwittingly been testing the default storage factory with the converted test.

          Show
          Cheng Che Chen added a comment - I have just emailed scanned images of signed ICLA to secretary@apache.org, as instructed by Kathey Marsden. Regarding Rick Hillegas' question about the tests included in my patch, there are three sets of unit tests and one functional test. The unit tests verify the correctness of my internal data structures. They are probably not useful to other implementations, unless, for example, they have similar properties to be verified. The functional test implements an application in both JAVA and SQL and cross-checks the computed results. It can indeed be applied to any implementation of storage factory by simply changing the database name for the SQL portion of the test object. The functional test consists of the following files: "GenMog.java", "SampMog.java", "ClusMog.java", "ClusMogSQL.java", and "MogTest.java" – the last file is the JUnit test object. However, I have some doubt about whether I have configured/used Derby's test framework properly in "MogTest.java". This file started out as a stand-alone JUnit test. When I was running it stand-alone, it would incur no disk activity. After I converted the file to run under Derby's test framework, the disk activity LED on my computer would go wild every time I ran it. I wonder if I have unwittingly been testing the default storage factory with the converted test.
          Hide
          Kristian Waagan added a comment -

          According to the 10.5 release page on the Wiki, the 10.5 branch will be
          created 2008-03-16. This leaves very little time to work on the
          in-memory back end. I'm describing my plan for this user-requested
          feature below. Most of the elements of the plan originates from the
          comments on this Jira issue.

            • Patch status
              The community has three patches available;
              A) derby-646-20090222.diff
              B) derby-646-2a-vfmem_first_rev.diff
              C) svn.diff

          Patch C is the original patch from Stephen Fitch. It was uploaded in
          March 2006. It is writing the database log to disk. The general state is
          not very well known, but I believe it is not quite "production ready"
          int it's current form. The patch demonstrates the overall approach,
          which is also used by the two other patches. I can't speak for A, but B
          is more or less a re-implementation of C. This follows from the
          interfaces that have to be implemented.
          Stephen Fitch has a CLA on file.

          Patch A is from Cheng Che Chen, which has written a solution of his own.
          The solution passes basic ad-hoc testing, and has been successfully used
          in several of the performance tests in the Derby code repository.
          This solution seems to be the most feature rich one, and it also has a
          functional test.
          Cheng Che Chen has a CLA on file.

          Patch B is from Kristian Waagan. The test has passed ad-hoc testing, and
          has been successfully used in several of the performance tests in the
          Derby code repository. The main difference from patch A, is that it
          seems to perform better on CMT machines.
          The solution is able to run most of suites.All without failures.
          Kristian Waagan has a CLA on file.

            • Plan
              1) Commit patch B.
              (reasons: CMT performance, size, readability)
              2) Work on the committed code, addressing the TODOs.
              3) Incorporate tests from patch A and some additional tests.
              (additional tests: backup in-mem then restore/re-boot with directory
              (default) storage engine, simple boot/create test)
              4) Incorporate features from patch A.
              (persist to disk on exit/shutdown, restore from disk on boot,
              finalizer, more?)
              5) Figure out what to do on OOME (if anything).
              6) General testing and improvements.
              7) Documentation.

          I hope to address at least steps 1-3 before the branch is cut.
          It is not clear to me whether an in-memory back end will be a documented
          and supported feature of 10.5. I like the idea of providing it as-is
          without any documentation in 10.5, and then solidify it in 10.6.
          In the meantime, interested users can try it out and give us feedback,
          and the community can continue to test and improve the feature.

          Unless someone objects, I will commit the patch on Monday.
          I would also like to invite those who plan to contribute actively on the
          feature to let the community know. Feel free to flesh out the details of
          the plan.

          Finally, a great thank you to both Stephen Fitch and Cheng Che Chen for
          contributing their work to the Derby community! I hope continued
          contribution is of interest
          I would be very happy if Cheng could spare some time to get some of his
          more advanced features into the Derby code line.

          Show
          Kristian Waagan added a comment - According to the 10.5 release page on the Wiki, the 10.5 branch will be created 2008-03-16. This leaves very little time to work on the in-memory back end. I'm describing my plan for this user-requested feature below. Most of the elements of the plan originates from the comments on this Jira issue. Patch status The community has three patches available; A) derby-646-20090222.diff B) derby-646-2a-vfmem_first_rev.diff C) svn.diff Patch C is the original patch from Stephen Fitch. It was uploaded in March 2006. It is writing the database log to disk. The general state is not very well known, but I believe it is not quite "production ready" int it's current form. The patch demonstrates the overall approach, which is also used by the two other patches. I can't speak for A, but B is more or less a re-implementation of C. This follows from the interfaces that have to be implemented. Stephen Fitch has a CLA on file. Patch A is from Cheng Che Chen, which has written a solution of his own. The solution passes basic ad-hoc testing, and has been successfully used in several of the performance tests in the Derby code repository. This solution seems to be the most feature rich one, and it also has a functional test. Cheng Che Chen has a CLA on file. Patch B is from Kristian Waagan. The test has passed ad-hoc testing, and has been successfully used in several of the performance tests in the Derby code repository. The main difference from patch A, is that it seems to perform better on CMT machines. The solution is able to run most of suites.All without failures. Kristian Waagan has a CLA on file. Plan 1) Commit patch B. (reasons: CMT performance, size, readability) 2) Work on the committed code, addressing the TODOs. 3) Incorporate tests from patch A and some additional tests. (additional tests: backup in-mem then restore/re-boot with directory (default) storage engine, simple boot/create test) 4) Incorporate features from patch A. (persist to disk on exit/shutdown, restore from disk on boot, finalizer, more?) 5) Figure out what to do on OOME (if anything). 6) General testing and improvements. 7) Documentation. I hope to address at least steps 1-3 before the branch is cut. It is not clear to me whether an in-memory back end will be a documented and supported feature of 10.5. I like the idea of providing it as-is without any documentation in 10.5, and then solidify it in 10.6. In the meantime, interested users can try it out and give us feedback, and the community can continue to test and improve the feature. Unless someone objects, I will commit the patch on Monday. I would also like to invite those who plan to contribute actively on the feature to let the community know. Feel free to flesh out the details of the plan. Finally, a great thank you to both Stephen Fitch and Cheng Che Chen for contributing their work to the Derby community! I hope continued contribution is of interest I would be very happy if Cheng could spare some time to get some of his more advanced features into the Derby code line.
          Hide
          Adrian Cole added a comment -

          JBoss Cache would consider switching onto Derby for some internal work, if this feature is implemented. As of now, our tests take 2 minutes longer with embedded derby then hsqldb due to file i/o.

          Show
          Adrian Cole added a comment - JBoss Cache would consider switching onto Derby for some internal work, if this feature is implemented. As of now, our tests take 2 minutes longer with embedded derby then hsqldb due to file i/o.
          Hide
          Kristian Waagan added a comment -

          I'm postponing the first commit until tomorrow, mostly because I've run out of time and want to write a overview of the code for the commit message.
          As soon as the patch is in, users are welcome to test it out.
          Note that the first revision won't have the features save on shutdown and restore on boot.

          Show
          Kristian Waagan added a comment - I'm postponing the first commit until tomorrow, mostly because I've run out of time and want to write a overview of the code for the commit message. As soon as the patch is in, users are welcome to test it out. Note that the first revision won't have the features save on shutdown and restore on boot.
          Hide
          Kristian Waagan added a comment -

          Committed patch 'derby-646-2b-vfmem_first_rev.diff' to trunk with revision 752114.

          I expect more work will be done under various sub-tasks of this issue.
          It would be nice to get early feedback from users and developers willing to try out the feature under development.
          Note that I have set the subSubProtocol to be 'in-memory' based on the feedback on DERBY-4084, but this may change.

          The in-memory back end is working in my environment, but there may be problems on other platforms. For instance, is the case-insensitive mode of Windows a problem?

          Show
          Kristian Waagan added a comment - Committed patch 'derby-646-2b-vfmem_first_rev.diff' to trunk with revision 752114. I expect more work will be done under various sub-tasks of this issue. It would be nice to get early feedback from users and developers willing to try out the feature under development. Note that I have set the subSubProtocol to be 'in-memory' based on the feedback on DERBY-4084 , but this may change. The in-memory back end is working in my environment, but there may be problems on other platforms. For instance, is the case-insensitive mode of Windows a problem?
          Hide
          Kristian Waagan added a comment -

          Indicating that the first iteration of this feature is targeted for 10.5.

          Show
          Kristian Waagan added a comment - Indicating that the first iteration of this feature is targeted for 10.5.
          Hide
          Kristian Waagan added a comment -

          Patch 3a is a JMX experiment (very poorly documented, not very well tested, contains bad code). It provides two JMX attributes and one operation; getDatabases, estimateSpaceUsage, and estimateSpaceUsage(String).
          You can obtain a list of the databases stored in memory, get a space usage estimate for all the databases or for a single database.
          Is this something we should consider adding?

          We could also allow for deleting databases through JMX, but then we have to do authentication (plus authorization?).

          Show
          Kristian Waagan added a comment - Patch 3a is a JMX experiment (very poorly documented, not very well tested, contains bad code). It provides two JMX attributes and one operation; getDatabases, estimateSpaceUsage, and estimateSpaceUsage(String). You can obtain a list of the databases stored in memory, get a space usage estimate for all the databases or for a single database. Is this something we should consider adding? We could also allow for deleting databases through JMX, but then we have to do authentication (plus authorization?).
          Hide
          Francois Orsini added a comment -

          +1. Having these (hooks) even as an experiment could be very useful while the in-memory feature is being tested.

          I would probably wait after 10.5 because allowing for database deletion, but that is just me - just because there needs probably more time to investigate it and its possible side effects.

          Maybe worth re-enforcing somewhere (notes if any) that the database is locked and its activity blocked while its estimated space usage is computed.

          Show
          Francois Orsini added a comment - +1. Having these (hooks) even as an experiment could be very useful while the in-memory feature is being tested. I would probably wait after 10.5 because allowing for database deletion, but that is just me - just because there needs probably more time to investigate it and its possible side effects. Maybe worth re-enforcing somewhere (notes if any) that the database is locked and its activity blocked while its estimated space usage is computed.
          Hide
          Knut Anders Hatlen added a comment -

          I came across one issue when I experimented with the in-memory backend. When I exit from ij, I see this exception:

          Exception in thread "main" java.lang.NullPointerException
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(BaseDataFileFactory.java:2684)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.removeStubs(BaseDataFileFactory.java:1591)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.stop(BaseDataFileFactory.java:499)
          at org.apache.derby.impl.services.monitor.TopService.stop(TopService.java:405)
          at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:349)
          at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:239)
          at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:205)
          at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
          at java.sql.DriverManager.getConnection(DriverManager.java:582)
          at java.sql.DriverManager.getConnection(DriverManager.java:207)
          at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:413)
          at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:246)
          at org.apache.derby.impl.tools.ij.Main.go(Main.java:210)
          at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:177)
          at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
          at org.apache.derby.tools.ij.main(ij.java:59)

          To reproduce, execute the following commands in ij:

          connect 'jdbc:derby:in-memory:mydb;create=true';
          exit;

          Other than that, it seems to be working fine.

          Show
          Knut Anders Hatlen added a comment - I came across one issue when I experimented with the in-memory backend. When I exit from ij, I see this exception: Exception in thread "main" java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(BaseDataFileFactory.java:2684) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.removeStubs(BaseDataFileFactory.java:1591) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.stop(BaseDataFileFactory.java:499) at org.apache.derby.impl.services.monitor.TopService.stop(TopService.java:405) at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:349) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:239) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:205) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:207) at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:413) at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:246) at org.apache.derby.impl.tools.ij.Main.go(Main.java:210) at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:177) at org.apache.derby.impl.tools.ij.Main.main(Main.java:73) at org.apache.derby.tools.ij.main(ij.java:59) To reproduce, execute the following commands in ij: connect 'jdbc:derby:in-memory:mydb;create=true'; exit; Other than that, it seems to be working fine.
          Hide
          Kristian Waagan added a comment -

          Thanks for the testing, Knut Anders.
          Seems like I messed up some of the path-handling logic (absolute vs relative) in the patch I committed. I'm looking into the problem now.

          Show
          Kristian Waagan added a comment - Thanks for the testing, Knut Anders. Seems like I messed up some of the path-handling logic (absolute vs relative) in the patch I committed. I'm looking into the problem now.
          Hide
          Cheng Che Chen added a comment -

          Kristian, you might want to adapt the functional test that I had submitted for your own in-memory storage factory. That way, you can have your changes automatically tested to some extent after every build.

          If I understand correctly, you tested your initial implementation by manually changing the default storage factory to point to your implementation and then running the Derby unit tests.

          As for helping out with developing this feature further, I'll see what I can do. In the coming weeks, I don't see much free time to devote to this. So perhaps for the release after the imminent upcoming one, there'll be an opportunity.

          Show
          Cheng Che Chen added a comment - Kristian, you might want to adapt the functional test that I had submitted for your own in-memory storage factory. That way, you can have your changes automatically tested to some extent after every build. If I understand correctly, you tested your initial implementation by manually changing the default storage factory to point to your implementation and then running the Derby unit tests. As for helping out with developing this feature further, I'll see what I can do. In the coming weeks, I don't see much free time to devote to this. So perhaps for the release after the imminent upcoming one, there'll be an opportunity.
          Hide
          Kristian Waagan added a comment -

          Cheng, thank you for your continued interest

          Since the code branch is cut on the coming Monday, there isn't much more that will be done for the in-memory solution for the upcoming 10.5 release, except testing and stabilization. This leaves plenty of time for an opportunity if you want to contribute later on.

          Regarding testing, I think that is a very good idea. I have already started extracting the Mog test from your patch, and it runs fine with the current code.
          The unit tests aren't as easy to adopt, as they test somewhat different characteristics / functionality (for instance peek, and mark/reset) and rely partly on internal variables etc. However, it shouldn't be hard to use your work to write similar tests for the current implementation.

          I'm sorry I'm charging on without waiting for more feedback from you and other interested parties. The reason is merely my wish to get something committed in time for 10.5. This will allow interested users to more easily test out the feature and give the community valuable feedback before the next feature release.

          I'm restating my goal for the feature:
          A basic, working in-memory storage back end that can be used in scenarios where durability isn't required;
          o basic, it won't have functionality like save on shutdown / JVM exit, or automatic restore on boot.
          o working, but not tested to the same degree as the rest of the Derby code.
          o undocumented (in the user docs).

          Show
          Kristian Waagan added a comment - Cheng, thank you for your continued interest Since the code branch is cut on the coming Monday, there isn't much more that will be done for the in-memory solution for the upcoming 10.5 release, except testing and stabilization. This leaves plenty of time for an opportunity if you want to contribute later on. Regarding testing, I think that is a very good idea. I have already started extracting the Mog test from your patch, and it runs fine with the current code. The unit tests aren't as easy to adopt, as they test somewhat different characteristics / functionality (for instance peek, and mark/reset) and rely partly on internal variables etc. However, it shouldn't be hard to use your work to write similar tests for the current implementation. I'm sorry I'm charging on without waiting for more feedback from you and other interested parties. The reason is merely my wish to get something committed in time for 10.5. This will allow interested users to more easily test out the feature and give the community valuable feedback before the next feature release. I'm restating my goal for the feature: A basic, working in-memory storage back end that can be used in scenarios where durability isn't required; o basic, it won't have functionality like save on shutdown / JVM exit, or automatic restore on boot. o working, but not tested to the same degree as the rest of the Derby code. o undocumented (in the user docs).
          Hide
          Myrna van Lunteren added a comment -

          So the initial commits are in 10.5. How can we proceed with this?
          New JIRA's for subsequent work?

          Show
          Myrna van Lunteren added a comment - So the initial commits are in 10.5. How can we proceed with this? New JIRA's for subsequent work?
          Hide
          Kristian Waagan added a comment -

          The core functionality is in 10.5 already.
          I'd like to get the following into 10.5 as well (DERBY-4085):
          a) Add the functional test written by Cheng.
          b) Enable the new tests as part of a suite (i.e. suites.All and as the ant junit-core target).

          New features will be logged as separate Jiras and added to the trunk.
          I do plan to write a Wiki-page documenting the feature to some extent. And then I hope users will give us feedback so that we can consider adding more features for the next Derby feature release.

          Show
          Kristian Waagan added a comment - The core functionality is in 10.5 already. I'd like to get the following into 10.5 as well ( DERBY-4085 ): a) Add the functional test written by Cheng. b) Enable the new tests as part of a suite (i.e. suites.All and as the ant junit-core target). New features will be logged as separate Jiras and added to the trunk. I do plan to write a Wiki-page documenting the feature to some extent. And then I hope users will give us feedback so that we can consider adding more features for the next Derby feature release.
          Hide
          Kathey Marsden added a comment -

          I think that until this feature is documented and fully supported, it shouldn't be resolved and be included in the release notes. The separate Jiras could be subtasks of this one and then the parent issue resolved for 10.6. Just my opinion.

          Show
          Kathey Marsden added a comment - I think that until this feature is documented and fully supported, it shouldn't be resolved and be included in the release notes. The separate Jiras could be subtasks of this one and then the parent issue resolved for 10.6. Just my opinion.
          Hide
          Myrna van Lunteren added a comment -

          Following Kathey's recommendation, taking off fix in 10.5 - so this won't be in the 10.5 fixed bugs list. It's still on the list of new functionality. We need to somehow point out that this is the first stage, though. Is it ok to say it's 'experimental' or is that too scary?

          Show
          Myrna van Lunteren added a comment - Following Kathey's recommendation, taking off fix in 10.5 - so this won't be in the 10.5 fixed bugs list. It's still on the list of new functionality. We need to somehow point out that this is the first stage, though. Is it ok to say it's 'experimental' or is that too scary?
          Hide
          Rick Hillegas added a comment -

          I don't think that we can say that this is a finished, full-fledged feature. However, I would like to see the release notes mention that this is an experimental feature which we'd like users to test-drive. We will need some kind of primer (perhaps a wiki page) and it would be great if the release notes could direct users to that primer for more information.

          Show
          Rick Hillegas added a comment - I don't think that we can say that this is a finished, full-fledged feature. However, I would like to see the release notes mention that this is an experimental feature which we'd like users to test-drive. We will need some kind of primer (perhaps a wiki page) and it would be great if the release notes could direct users to that primer for more information.
          Show
          Kristian Waagan added a comment - Primer at http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
          Hide
          Knut Magne Solem added a comment -

          Good to see something is happening in this field! Out of curiosity I tried this out on Windows XP, but it failed in embedded mode when creating an in-memory database.

          The error was this assert:
          ERROR XJ001: Java exception: 'ASSERT FAILED serviceName = memory:C:\Documents and Settings\user\workspace\derby\MyDbTest;storageFactory.getCanonicalName() = C:\Documents and Settings\user\workspace\derby\MyDbTest: org.apache.derby.shared.common.sanity.AssertFailure'.

          To make it work I altered line 310 in VFMemoryStorageFactory.normalizePath() to allow windows-style paths. I changed it to this: "} else if (dir.charAt(0) != getSeparator() && !dir.toUpperCase().matches("[a-zA-Z]

          {1}:{1}

          \\\\

          {1}

          .*")){"

          Appart from that i worked great.

          To reduce memory footprint I guess you can reduce the cache to a minimum and maybe discard flushed log records instead of writing them to memory?

          Knut Magne,
          (previous user knutmas)

          Show
          Knut Magne Solem added a comment - Good to see something is happening in this field! Out of curiosity I tried this out on Windows XP, but it failed in embedded mode when creating an in-memory database. The error was this assert: ERROR XJ001: Java exception: 'ASSERT FAILED serviceName = memory:C:\Documents and Settings\user\workspace\derby\MyDbTest;storageFactory.getCanonicalName() = C:\Documents and Settings\user\workspace\derby\MyDbTest: org.apache.derby.shared.common.sanity.AssertFailure'. To make it work I altered line 310 in VFMemoryStorageFactory.normalizePath() to allow windows-style paths. I changed it to this: "} else if (dir.charAt(0) != getSeparator() && !dir.toUpperCase().matches(" [a-zA-Z] {1}:{1} \\\\ {1} .*")){" Appart from that i worked great. To reduce memory footprint I guess you can reduce the cache to a minimum and maybe discard flushed log records instead of writing them to memory? Knut Magne, (previous user knutmas)
          Hide
          Kristian Waagan added a comment -

          Thanks for testing the new feature, Knut Magne
          I'm currently working on a patch, where I'm considering using java.io.File. That should take care of any other OS that has a different style for paths as well.
          Feel free to log a bug in Jira. If you don't, I will tomorrow.

          Regarding memory footprint, you have to balance the memory overhead against performance. Setting the page cache to the minimum size will degrade performance in many configurations.
          The worst case I saw in my simple testing, was a drop from 28361 (pageCacheSize=1000) tps to 4059 (pageCacheSize=40) tps. This was a single record update test using a page size of 32 K.

          I'm not quite sure how you would go about discarding the log records, but you do need the UNDO log records to support transactions. Doing something in this area should be possible.
          I don't know the log system well enough to say something more, but maybe someone else does?

          Show
          Kristian Waagan added a comment - Thanks for testing the new feature, Knut Magne I'm currently working on a patch, where I'm considering using java.io.File. That should take care of any other OS that has a different style for paths as well. Feel free to log a bug in Jira. If you don't, I will tomorrow. Regarding memory footprint, you have to balance the memory overhead against performance. Setting the page cache to the minimum size will degrade performance in many configurations. The worst case I saw in my simple testing, was a drop from 28361 (pageCacheSize=1000) tps to 4059 (pageCacheSize=40) tps. This was a single record update test using a page size of 32 K. I'm not quite sure how you would go about discarding the log records, but you do need the UNDO log records to support transactions. Doing something in this area should be possible. I don't know the log system well enough to say something more, but maybe someone else does?
          Hide
          Knut Magne Solem added a comment -

          You need the log for undo, but for committed transactions I can't see why you need to keep the log records for a non-durable in-memory database. It could be hard to do this because you are doing in-memory stuff on this level. I tried to just discard writing log records to virtualFiles, but rollback failed because it was flushing the log and reading undo records from the storage. So we need to keep some log in virtualFiles to get undo working

          Show
          Knut Magne Solem added a comment - You need the log for undo, but for committed transactions I can't see why you need to keep the log records for a non-durable in-memory database. It could be hard to do this because you are doing in-memory stuff on this level. I tried to just discard writing log records to virtualFiles, but rollback failed because it was flushing the log and reading undo records from the storage. So we need to keep some log in virtualFiles to get undo working
          Hide
          Kristian Waagan added a comment -

          Added link for the issue tracking the addition of user documentation for the in-memory back end.

          Show
          Kristian Waagan added a comment - Added link for the issue tracking the addition of user documentation for the in-memory back end.
          Hide
          Knut Anders Hatlen added a comment -

          It seems like the fix for this issue went into 10.5. Could we close it now?

          Show
          Knut Anders Hatlen added a comment - It seems like the fix for this issue went into 10.5. Could we close it now?
          Hide
          Kristian Waagan added a comment -

          Closing issue.
          Further improvements and additions should be tracked under separate Jiras.

          Show
          Kristian Waagan added a comment - Closing issue. Further improvements and additions should be tracked under separate Jiras.

            People

            • Assignee:
              Kristian Waagan
              Reporter:
              Stephen Fitch
            • Votes:
              37 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development