Derby
  1. Derby
  2. DERBY-2852

Derby: Replication and Availability

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 10.2.2.0
    • Fix Version/s: None
    • Component/s: Network Server, Services, SQL, Store
    • Labels:
      None

      Description

      This paper and attached source code describes the work done to add hot standby replication functionality to the Apache Derby Database Management System.

      By implementing a hot standby scheme in Apache Derby several features are added. The contents of the database is replicated at run time to another site providing online runtime backup. As the hot standby takes over on faults availability is added in that a client can connect to the hot standby after a crash. Thus the crash is masked from the clients. In addition to this, online upgrades of software and hardware can be done by taking down one database at the time. Then when
      the upgrade is completed the upgraded server is synchronized and back online with no downtime.

      A fully functional prototype of the Apache Derby hot standby scheme has been created in this project using logical logs, fail-fast takeovers and logical catchups after an internal up-to-crash recovery and reconnection. This project builds on the ideas that are presented in my earlier project: Derby: Write to Neighbor Mode.

      To enable the features described above some changes have been made to the Derby base system. The following classes have been changed or added:

      The Logical Log System
      org.apache.derby.iapi.store.logical.log.LogicalImage
      org.apache.derby.iapi.store.logical.log.LogicalLoggable
      org.apache.derby.impl.store.logical.log.LogicalLog
      org.apache.derby.impl.store.logical.log.LogicalLogger
      org.apache.derby.impl.store.logical.log.LogicalLogRecord
      org.apache.derby.impl.store.logical.opearations.LogicalAbortOperation
      org.apache.derby.impl.store.logical.opearations.LogicalCommitOperation
      org.apache.derby.impl.store.logical.opearations.LogicalDeleteOperation
      org.apache.derby.impl.store.logical.opearations.LogicalImage
      org.apache.derby.impl.store.logical.opearations.LogicalInsertOperation
      org.apache.derby.impl.store.logical.opearations.LogicalOperation
      org.apache.derby.impl.store.logical.opearations.LogicalOperationFactory
      org.apache.derby.impl.store.logical.opearations.LogicalUpdateOperation

      The SQL Layer
      org.apache.impl.sql.execute.RowChangerImpl
      org.apache.impl.sql.conn.GenericLanguageConnectionContext

      The Communications Service
      org.apache.derby.impl.services.net.LogicalServer
      org.apache.derby.impl.services.net.LogicalClient
      org.apache.derby.impl.services.net.NetworkPayload
      org.apache.derby.impl.services.net.receiver.LogReceiver
      org.apache.derby.impl.services.net.shipper.LogShipper
      org.apache.derby.impl.services.net.shipper.LogicalCatchUp

      The Hot Standby Service
      org.apache.derby.impl.services.hotstandby.HotStandbyController
      org.apache.derby.impl.services.hotstandby.Database

      The Network Server
      org.apache.derby.impl.drda.NetworkServerControlImpl
      org.apache.derby.impl.drda.HotStandbyPoller
      org.apache.derby.impl.drda.net.LogicalClient
      org.apache.derby.imp.drda.net.LogicalServer

      See the added paper for more thorough documentation on design choices and implementation along with some benchmarking results

      1. svn.diff
        32 kB
        Egil Sørensen
      2. soereeg07
        1.33 MB
        Egil Sørensen
      3. repl_trunk.stat
        3 kB
        Jørgen Løland
      4. repl_trunk.diff
        124 kB
        Jørgen Løland
      5. repl_trunk_ignored.diff
        8 kB
        Jørgen Løland
      6. derby-rep.zip
        91 kB
        Egil Sørensen

        Issue Links

          Activity

          Hide
          Kristian Waagan added a comment -

          Thanks for the feedback, Rick.
          Resolving this a a duplicate of DERBY-2872 and closing the issue.

          Show
          Kristian Waagan added a comment - Thanks for the feedback, Rick. Resolving this a a duplicate of DERBY-2872 and closing the issue.
          Hide
          Rick Hillegas added a comment -

          I haven't looked at the patch. It's possible that it contains some valuable functionality (like automatic failover?) on top of the replication which was actually added by DERBY-2872. Since this issue is linked to DERBY-2872, I think we'll be able to find that additional functionality if we need to. I see no need to keep this issue open. Thanks.

          Show
          Rick Hillegas added a comment - I haven't looked at the patch. It's possible that it contains some valuable functionality (like automatic failover?) on top of the replication which was actually added by DERBY-2872 . Since this issue is linked to DERBY-2872 , I think we'll be able to find that additional functionality if we need to. I see no need to keep this issue open. Thanks.
          Hide
          Kristian Waagan added a comment -

          It's been a long time, but does anyone know the present state of this issue?
          From what I can see, there appears to be an ICLA on file ([1]), although I'm not 100% sure it's for the right person.

          However, replication has already been added (see DERBY-2872). Should this issue be closed, or is it still valuable in some sense and should remain open?

          Thanks,

          [1] http://people.apache.org/committer-index.html

          Show
          Kristian Waagan added a comment - It's been a long time, but does anyone know the present state of this issue? From what I can see, there appears to be an ICLA on file ( [1] ), although I'm not 100% sure it's for the right person. However, replication has already been added (see DERBY-2872 ). Should this issue be closed, or is it still valuable in some sense and should remain open? Thanks, [1] http://people.apache.org/committer-index.html
          Hide
          Egil Sørensen added a comment -

          I am terribly sorry that I have not yet submitted the ICLA, I have signed it already, but had to wait for a final clearance from the University that the source code was mine to donate. I will send it tomorrow, you should have it within the week.

          Sorry for the inconvenience.

          Show
          Egil Sørensen added a comment - I am terribly sorry that I have not yet submitted the ICLA, I have signed it already, but had to wait for a final clearance from the University that the source code was mine to donate. I will send it tomorrow, you should have it within the week. Sorry for the inconvenience.
          Hide
          Jørgen Løland added a comment -

          Unchecking Patch Available.

          The code is an excellent proof of concept for the functionality (after all, it works ), but after studying it I do think there is more work that needs to be done for it to be considered for inclusion in Derby.

          Egil: feel free to change to "patch available" again if you do not agree but think the patch should be considered for inclusion as it is.

          Show
          Jørgen Løland added a comment - Unchecking Patch Available. The code is an excellent proof of concept for the functionality (after all, it works ), but after studying it I do think there is more work that needs to be done for it to be considered for inclusion in Derby. Egil: feel free to change to "patch available" again if you do not agree but think the patch should be considered for inclusion as it is.
          Hide
          Egil Sørensen added a comment -

          I have planned to submit an ICLA, I intended to do so, but due to my vacation it was forgotten. Hopefully it will be submitted within the next 3 or 4 days.

          Show
          Egil Sørensen added a comment - I have planned to submit an ICLA, I intended to do so, but due to my vacation it was forgotten. Hopefully it will be submitted within the next 3 or 4 days.
          Hide
          Daniel John Debrunner added a comment -

          Egil, are you planning to submit an ICLA to Apache?
          If not, there is the significant chance that this patch cannot be accepted into the Derby codeline.

          Show
          Daniel John Debrunner added a comment - Egil, are you planning to submit an ICLA to Apache? If not, there is the significant chance that this patch cannot be accepted into the Derby codeline.
          Hide
          Egil Sørensen added a comment -

          Ah, the java.io.FileNotFoundException: logicallog/logical.log (No such file or directory) is of course due to the folder 'logicallog' missing from the path the database is run from. To fix this, either create a folder named logicallog or replace the path in org.apache.derby.impl.store.logical.logicallog#flushToDisk

          I'm sorry for not informing you about this. Thanks!

          Show
          Egil Sørensen added a comment - Ah, the java.io.FileNotFoundException: logicallog/logical.log (No such file or directory) is of course due to the folder 'logicallog' missing from the path the database is run from. To fix this, either create a folder named logicallog or replace the path in org.apache.derby.impl.store.logical.logicallog#flushToDisk I'm sorry for not informing you about this. Thanks!
          Hide
          Jørgen Løland added a comment -

          Egil,

          I think the posted replication functionality looks really interesting. I have had a quick look at your code, and tried to make a patch for the current trunk version to make it easier for other developers to test it.

          repl_trunk.* contains the patch, while repl_trunk_ignored.diff contains changes in the originally posted diff that I believe should not be part of the replication patch (maybe noise from diff'ing between two different derby revisions?)

          Note that the attached files are almost equal to what was posted by Egil in the zip and diff files. I have only made some small changes to make it fit into trunk. Furthermore, the files that were in the zip file (added files) are now found in the diff file.

          I have had a few test runs and have encountered a few of these on the Derby running as "primary":
          java.io.FileNotFoundException: logicallog/logical.log (No such file or directory)

          No debugging has been performed on this.

          Show
          Jørgen Løland added a comment - Egil, I think the posted replication functionality looks really interesting. I have had a quick look at your code, and tried to make a patch for the current trunk version to make it easier for other developers to test it. repl_trunk.* contains the patch, while repl_trunk_ignored.diff contains changes in the originally posted diff that I believe should not be part of the replication patch (maybe noise from diff'ing between two different derby revisions?) Note that the attached files are almost equal to what was posted by Egil in the zip and diff files. I have only made some small changes to make it fit into trunk. Furthermore, the files that were in the zip file (added files) are now found in the diff file. I have had a few test runs and have encountered a few of these on the Derby running as "primary": java.io.FileNotFoundException: logicallog/logical.log (No such file or directory) No debugging has been performed on this.
          Hide
          Daniel John Debrunner added a comment -

          Very interesting, I assume the diffs are against 10.2.2.0. Have you tried applying it to the trunk, that's where it will need to go if it gets committed.

          Before the Derby project can accept this you need to file an ICLA with the ASF

          These pages on the Derby wiki has information for getting involved in the Derby project.

          http://wiki.apache.org/db-derby/DerbyDev
          http://wiki.apache.org/db-derby/DerbyContributorChecklist

          Show
          Daniel John Debrunner added a comment - Very interesting, I assume the diffs are against 10.2.2.0. Have you tried applying it to the trunk, that's where it will need to go if it gets committed. Before the Derby project can accept this you need to file an ICLA with the ASF These pages on the Derby wiki has information for getting involved in the Derby project. http://wiki.apache.org/db-derby/DerbyDev http://wiki.apache.org/db-derby/DerbyContributorChecklist
          Hide
          Egil Sørensen added a comment -

          The svn diff from current release (10.2)

          Show
          Egil Sørensen added a comment - The svn diff from current release (10.2)
          Hide
          Egil Sørensen added a comment -

          Added and changed source code

          Show
          Egil Sørensen added a comment - Added and changed source code
          Hide
          Egil Sørensen added a comment -

          Documentation on the logical replication scheme added to Apache Derby

          Show
          Egil Sørensen added a comment - Documentation on the logical replication scheme added to Apache Derby

            People

            • Assignee:
              Unassigned
              Reporter:
              Egil Sørensen
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development