Derby
  1. Derby
  2. DERBY-3533

Replication fails when unlogged operations are used

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.6.1.0
    • Fix Version/s: 10.6.1.0
    • Component/s: Replication
    • Labels:
      None
    • Bug behavior facts:
      Crash

      Description

      I created an index while replication was running, and the slave seems to have failed at that time. Below is excerpts from derby.log on the slave. The container id referred to, matches the id of the index on the master.

      ------------ BEGIN SHUTDOWN ERROR STACK -------------

      ---- BEGIN REPLICATION ERROR MESSAGE (3/11/08 2:32 PM) ----
      ERROR XSLA7: Cannot redo operation Page Operation: Page(349,Container(0, 1041)) pageVersion 180 : Insert : Slot=177 recordId=183 in the log.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:296)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1525)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: ERROR XSDG0: Page Page(349,Container(0, 1041)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:688)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      ... 17 more
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      ... 25 more
      ============= begin nested exception, level (1) ===========
      ERROR XSDG0: Page Page(349,Container(0, 1041)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:688)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      ... 25 more
      ============= end nested exception, level (1) ===========
      ============= begin nested exception, level (2) ===========
      java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      ============= end nested exception, level (2) ===========

      ------------ END SHUTDOWN ERROR STACK -------------

      Replication slave role was stopped for database 'slaveDB'.
      Replication slave got a fatal error for database 'slaveDB'. Replication will be stopped.
      ERROR XSLAA: The store has been marked for shutdown by an earlier exception.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:279)
      at org.apache.derby.impl.store.raw.log.LogToFile.appendLogRecord(LogToFile.java:3704)
      at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.handleLogChunk(SlaveController.java:535)
      at org.apache.derby.impl.store.replication.slave.SlaveController$SlaveLogReceiverThread.run(SlaveController.java:481)
      Caused by: ERROR XSLA7: Cannot redo operation Page Operation: Page(349,Container(0, 1041)) pageVersion 180 : Insert : Slot=177 recordId=183 in the log.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:296)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1525)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: ERROR XSDG0: Page Page(349,Container(0, 1041)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:688)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      ... 17 more
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      ... 25 more
      ============= begin nested exception, level (1) ===========
      ERROR XSLA7: Cannot redo operation Page Operation: Page(349,Container(0, 1041)) pageVersion 180 : Insert : Slot=177 recordId=183 in the log.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:296)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1525)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: ERROR XSDG0: Page Page(349,Container(0, 1041)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:688)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      ... 17 more
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      ... 25 more
      ============= end nested exception, level (1) ===========
      ============= begin nested exception, level (2) ===========
      ERROR XSDG0: Page Page(349,Container(0, 1041)) could not be read from disk.
      at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:688)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      ... 25 more
      ============= end nested exception, level (2) ===========
      ============= begin nested exception, level (3) ===========
      java.io.EOFException: Reached end of file while attempting to read a whole page.
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:378)
      at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:201)
      at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:664)
      at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
      at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:288)
      at org.apache.derby.impl.store.raw.data.FileContainer.getAnyPage(FileContainer.java:2493)
      at org.apache.derby.impl.store.raw.data.BaseContainer.getAnyPage(BaseContainer.java:496)
      at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getAnyPage(BaseContainerHandle.java:611)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.findpage(PageBasicOperation.java:304)
      at org.apache.derby.impl.store.raw.data.PageBasicOperation.needsRedo(PageBasicOperation.java:160)
      at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
      at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
      at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
      at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:789)
      at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:205)
      at org.apache.derby.impl.db.SlaveDatabase.bootBasicDatabase(SlaveDatabase.java:421)
      at org.apache.derby.impl.db.SlaveDatabase.access$000(SlaveDatabase.java:70)
      at org.apache.derby.impl.db.SlaveDatabase$SlaveDatabaseBootThread.run(SlaveDatabase.java:308)
      at java.lang.Thread.run(Thread.java:619)
      ============= end nested exception, level (3) ===========

      -------------------- END REPLICATION ERROR MESSAGE ---------------------
      Replication slave role was stopped for database 'slaveDB'.

        Activity

        Hide
        Jørgen Løland added a comment -

        I don't understand why I haven't provoked this before since I have tried to create indexes on replicated databases. Maybe I haven't inserted enough data afterwards.

        Anyway - It turns out that some operations are not logged by default in derby. Both create index and import to an empty table are unlogged, as well as loading of jar files. Logging needs to be enabled before we copy the database to the slave location to solve the former two. Loading of jars will not be solved by this, however.

        The online backup implementation seems to have much of the code needed for this. There is e.g. a method blockBackupBlockingOperations in XactFactory that seems useful.

        I suggest that the replication step in which the master database is frozen is replaced by a new system procedure to solve index and import:

        Old: SYSCS_UTIL.SYSCS_FREEZE_DATABASE()
        New: SYSCS_UTIL.SYSCS_PREPARE_REPLICATION()

        The new system procedure should:
        1) Freeze the database
        2) Check if there are any ongoing transactions with unlogged operations. If so - unfreeze and abort. Otherwise:
        3) Enable logging of unlogged operations

        Regarding jars, I wonder whether we should add clear documentation or reject jar loading while replicated. Maybe there are more alternatives as well...

        Show
        Jørgen Løland added a comment - I don't understand why I haven't provoked this before since I have tried to create indexes on replicated databases. Maybe I haven't inserted enough data afterwards. Anyway - It turns out that some operations are not logged by default in derby. Both create index and import to an empty table are unlogged, as well as loading of jar files. Logging needs to be enabled before we copy the database to the slave location to solve the former two. Loading of jars will not be solved by this, however. The online backup implementation seems to have much of the code needed for this. There is e.g. a method blockBackupBlockingOperations in XactFactory that seems useful. I suggest that the replication step in which the master database is frozen is replaced by a new system procedure to solve index and import: Old: SYSCS_UTIL.SYSCS_FREEZE_DATABASE() New: SYSCS_UTIL.SYSCS_PREPARE_REPLICATION() The new system procedure should: 1) Freeze the database 2) Check if there are any ongoing transactions with unlogged operations. If so - unfreeze and abort. Otherwise: 3) Enable logging of unlogged operations Regarding jars, I wonder whether we should add clear documentation or reject jar loading while replicated. Maybe there are more alternatives as well...
        Hide
        V.Narayanan added a comment - - edited

        Thank you Jorgen, for summarizing the issue, suggesting a crisp explanation of the possible solution,
        code pointers to places from were code can be reused for this issue and issues that you think would
        form a highlight in this implementation.

        The series of steps (outlined above in the comments by Jorgen) if followed before replication startup
        on the master (done as a system procedure) will help to take care of the all the logged operations except
        jar file operations.

        I believe there is no issue with the index and the import operations, and that enabling logging
        for the unlogged operations will solve the problem encountered in these cases. * I however did not
        find much literature on the nature of the logs for these operations and would appreciate any inputs
        from the community here.*

        Jar Files
        ---------

        With regards to Jar files

        • The installed jar files are not logged but system catalog updates are logged
          when jar files are added/deleted.
        • This would mean that allowing jar file operations during deletion would result
          in the system catalog table on the slave having a reference to the jar file that
          does not exist in the backup database

        solution
        --------

        • Block jar file operations when backup is in progress
        • write clear documentation stating that the jar files will have to be
          manually copied to the slave.

        I intend to bifurcate this issue into two JIRA's

        1) Implement procedure SYSCS_UTIL.SYSCS_PREPARE_REPLICATION()
        2) Handle installed jar files during replication

        Show
        V.Narayanan added a comment - - edited Thank you Jorgen, for summarizing the issue, suggesting a crisp explanation of the possible solution, code pointers to places from were code can be reused for this issue and issues that you think would form a highlight in this implementation. The series of steps (outlined above in the comments by Jorgen) if followed before replication startup on the master (done as a system procedure) will help to take care of the all the logged operations except jar file operations. I believe there is no issue with the index and the import operations, and that enabling logging for the unlogged operations will solve the problem encountered in these cases. * I however did not find much literature on the nature of the logs for these operations and would appreciate any inputs from the community here.* Jar Files --------- With regards to Jar files The installed jar files are not logged but system catalog updates are logged when jar files are added/deleted. This would mean that allowing jar file operations during deletion would result in the system catalog table on the slave having a reference to the jar file that does not exist in the backup database solution -------- Block jar file operations when backup is in progress write clear documentation stating that the jar files will have to be manually copied to the slave. I intend to bifurcate this issue into two JIRA's 1) Implement procedure SYSCS_UTIL.SYSCS_PREPARE_REPLICATION() 2) Handle installed jar files during replication
        Hide
        V.Narayanan added a comment -

        Derby-3551 and Derby-3552 which are sub-tasks of this issue have
        been resolved. However Derby-3553 is still open. Hence marking this
        issue as affecting both 10.4 (due to Derby-3551 and Derby-3552)
        and 10.5 (Derby-3553)

        Show
        V.Narayanan added a comment - Derby-3551 and Derby-3552 which are sub-tasks of this issue have been resolved. However Derby-3553 is still open. Hence marking this issue as affecting both 10.4 (due to Derby-3551 and Derby-3552) and 10.5 (Derby-3553)
        Hide
        Tiago R. Espinha added a comment -

        Triaged for 10.5.2.
        Checked Crash, but the issue itself seems to be fixed and the remaining subtask is just to implement tests. I'm closing the issue as the bug itself is fixed.

        Show
        Tiago R. Espinha added a comment - Triaged for 10.5.2. Checked Crash, but the issue itself seems to be fixed and the remaining subtask is just to implement tests. I'm closing the issue as the bug itself is fixed.

          People

          • Assignee:
            V.Narayanan
            Reporter:
            Øystein Grøvlen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development