[QPID-7021] BDB hot backup (takeBackupNoLock) may fail with IOException

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.18, 0.22, 0.30, 0.32, qpid-java-6.0
• Fix Version/s: qpid-java-6.0.1, qpid-java-6.1
• Component/s: Broker-J
• Assignee: Lorenz Quack
• Reporter: Keith Wall
• Created: 25/Jan/16 15:06
• Updated: 02/Oct/19 19:21
• Resolved: 26/Jan/16 11:04

Description

The BDB hot backup script may fail with the following exception.

The program is flawed in that it believes acquiring file handles for the JDB files is sufficient to ensure that a subsequent separate open will succeed. This is not true, at least on any UNIX like file system I have seen. Having an open file e handle does not prevent another process removing the entry from the filesystem.

Starting Hot Backup Script
BDBBackup Utility: Starting Hot Backup.
java.io.FileNotFoundException: /home/qpid/prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory) fromDir:/local/qpid/prod/bdbstore/myvhost toDir:/local/qpid/bdbstore-backup/prod/QPIDBDBBackup_2016-01-25-11.20.32/bdbstore
java.io.FileNotFoundException: /local/qpid/prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.apache.qpid.util.FileUtils.copyCheckedEx(FileUtils.java:203)
        at org.apache.qpid.util.FileUtils.copy(FileUtils.java:185)
        at org.apache.qpid.server.store.berkeleydb.BDBBackup.takeBackupNoLock(BDBBackup.java:305)
        at org.apache.qpid.server.store.berkeleydb.BDBBackup.main(BDBBackup.java:126)
Backup script encountered an error and has failed: java.io.FileNotFoundException: /local/qpid/prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory) fromDir:/local/qpid/Prod/bdbstore/myvhost toDir:/local/qpid/bdbstore-backup/prod/QPIDBDBBackup_2016-01-25-11.20.32/bdbstore
Backup script got exception: java.io.FileNotFoundException: /local/qpid/Prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory) fromDir:/local/qpid/Prod/bdbstore/myvhost toDir:/local/qpid/bdbstore-backup/Prod/QPIDBDBBackup_2016-01-25-11.20.32/bdbstore
org.apache.qpid.server.store.StoreException: java.io.FileNotFoundException: /local/qpid/prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory) fromDir:/local/qpid/prod/bdbstore/myvhost toDir:/local/qpid/bdbstore-backup/Prod/QPIDBDBBackup_2016-01-25-11.20.32/bdbstore
        at org.apache.qpid.server.store.berkeleydb.BDBBackup.takeBackupNoLock(BDBBackup.java:312)
        at org.apache.qpid.server.store.berkeleydb.BDBBackup.main(BDBBackup.java:126)
Caused by: java.io.FileNotFoundException: /local/qpid/prod/bdbstore/myvhost/004d1f64.jdb (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.apache.qpid.util.FileUtils.copyCheckedEx(FileUtils.java:203)
        at org.apache.qpid.util.FileUtils.copy(FileUtils.java:185)
        at org.apache.qpid.server.store.berkeleydb.BDBBackup.takeBackupNoLock(BDBBackup.java:305)
        ... 1 more