1. Derby
  2. DERBY-2798

A new approach for main-memory database


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s:
    • Fix Version/s: None
    • Component/s: Store
    • Labels:
    • Environment:


      As a part of my Masters degree I have created an extension that allows data to reside in memory without the need to serialize it to Page-objects. This is a pretty big chunk of code and is sort of a proof of concept of another way to make an in-memory storage mode.

      I created two new conglomerates, called MemHeap and MemSkiplist. Derby interfaces them the same way as it does with Heap and BTree. These new conglomerates use RawStore for transaction support and logging, but not for storage. Instead it uses a new system service I've called MemStore. This data store only stores pointers to Slot-objects organized in arrays corresponding to its container/conglomerate/table. A Slot-object consists mainly of a DataValueDescriptor[]-object representing a row in a table.

      So, instead of just doing dummy-IO in memory where Derby still thinks its doing real IO and page caching, this new approach bypasses the cache and page-structure by keeping the DataValueDescriptor[]-objects in memory without serializing them.

      Manipulating operations on data in memory are done via new operation-objects (ex. MemInsertOperation, MemInsertUndoOperation, MemDeleteOperation...) with still uses RawStore for transaction control and persistence. Checkpointing is done by serializing the objects in MemStore fuzzily and completely unsynchronized to disk. Recovery consists of de-serializing the objects to MemStore before the existing REDO- and UNDO-phase of Derby recovery in RawStore will get the data transaction-consistent by replaying or undo the new operation-objects in the log.

      Locking is hard coded as row based with locking degree SERIALIZABLE.

      To get Derby to use the new conglomerates I hacked the SQL-layer to create MemHeap-tables and MemSkiplist-indexes when the table name starts with 'mem_'.

      Because this is a major rewrite of the access- and storage-layer there is a lot of known and unknown bugs and missing functionality. What is working is essentially select, insert, update and delete on tables with one primary key.

      1. Derby-
        368 kB
        Knut Magne Solem
      2. DERBY-2798.diff
        368 kB
        Knut Magne Solem
      3. DERBY-2798-
        340 kB
        Knut Magne Solem
      4. DERBY-2798-
        5 kB
        Knut Magne Solem
      5. select.png
        3 kB
        Knut Magne Solem
      6. update.png
        3 kB
        Knut Magne Solem

        Issue Links


          Knut Magne Solem created issue -
          Knut Magne Solem made changes -
          Field Original Value New Value
          Attachment Derby- [ 12359331 ]
          Knut Magne Solem made changes -
          Fix Version/s [ 10992 ]
          Fix Version/s [ 12312027 ]
          Knut Magne Solem made changes -
          Attachment DERBY-2798.diff [ 12359418 ]
          Myrna van Lunteren made changes -
          Fix Version/s [ 12312027 ]
          Kathey Marsden made changes -
          Link This issue is related to DERBY-646 [ DERBY-646 ]
          Knut Magne Solem made changes -
          Attachment select.png [ 12360430 ]
          Attachment DERBY-2798- [ 12360429 ]
          Attachment DERBY-2798- [ 12360428 ]
          Knut Magne Solem made changes -
          Attachment update.png [ 12360431 ]
          Bastian Wassermann made changes -
          Comment [ Sorry i have so many problems, getting this patch to run, so i have to post here. I'm very rooky in this whole patching thing.

          After i patched the release, an error occured that there is a module specification of derby.module.memstore is needed. So I added "cloudscape.config.memstore=all" in line 300 in After that change the compile with ant works, but i cant create a valid derby.jar file (it builds all the jar files, but the derby jar is only of 87kb size and only contains .properties (60 files) s.o., but no class files). Does someone know what i have done wrong, and what i have to change.

          The java version is 1.4.2 and i think ant is 1.6.2. ]
          Dyre Tjeldvoll made changes -
          Derby Info [Patch Available]
          Dag H. Wanvik made changes -
          Issue Type New Feature [ 2 ] Improvement [ 4 ]
          Rick Hillegas made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Won't Fix [ 2 ]
          Gavin made changes -
          Workflow jira [ 12405944 ] Default workflow, editable Closed status [ 12801551 ]


            • Assignee:
              Knut Magne Solem
            • Votes:
              4 Vote for this issue
              7 Start watching this issue


              • Created: