Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
backup is in the progress on the server.
Description
Currently Derby allows users to perfoms online backups using SYSCS_UTIL.SYSCS_BACKUP_DATABASE() procedure, but while the backup is in progress, update operations are temporarily blocked, but read operations can still proceed.
Blocking update operations can be real issue specifically in client server environments, because user requests will be blocked for a long time if a
backup is in the progress on the server.
phase 1 work for DERBY-239, committed on behalf of: Suresh Thalamati
This patch adds some code necessary to support real-time online backup that
does not block writes when database backup is in progress. All the current functional tests
passed with these changes. It would be great if some can review and commit this patch.
This patch changes the way data segment and log is backed up without blocking
the writes.
Data Segment Backup:
o The containers to be backed up are found by scanning the files in seg0.
o Each container is backed up by reading all the pages through the page cache,
and then writing to the backup container. Pages are latched while
writing into the backup container.
o Not necessary to backup containers in any particular order , All updates
that happens after a container is backed will be redone using the
transaction log on restore.
MT cases:
1) Each page is latched when it is written to the backup to prevent partial
written pages sneaking into the backup.
2) Thread that is backing up the container will stop if another thread
requests removal of the container when container is being backed up.
3) Truncate of the container blocks if the container is being backed up.
4) Partially created containers will not be backed up. Container cache will
not return the container items until the creation is complete.
(No changes are not for this case , that is how it currently works).
Transaction Log Backup:
Transaction Log file backup in two phases:
1) First Check point info and the the log files are backed up before the
data segment.
2) After the data segment is backed up , all the log files
that are generated after tha backup started are also copied into the backup.
MT cases:
1) If there is a checkpoint in progress, backup will wait for the
checkpoint to complete before copying checkpoint control information
into the backup.
anther patch for DERBY-239, committed on behalf of: Suresh Thalamati.
Fix to the problem found by Oystein while reviewing the previous online
backup patch(online_backup1.diff).
Backup of a container code was doing a seek incorrectly on the file container
instead of the backup file.
DERBY-239, committing next phase checkin on behalf of Suresh Thalamati.
This patch adds code to support real-time online backup with unlogged
operations. A consistent backup can not be made if there are pending
transactions with unlogged operations or if unlogged operations
occur when backup is in progress. Because container files can be copied
to the backup before the transaction is committed and the data pages are
flushed as part of the commit. As there is no transaction log for unlogged
operations, while restoring from the backup database can not be restored to
a consistent state. To make a consistent online backup in this scenario, this patch:
1) blocks online backup until all the transactions with unlogged operation are
committed/aborted.
2) implicitly converts all unlogged operations to logged mode for the duration
of the online backup, if they are started when backup is in progress.
This patch also adds a test to test the online backup in parallel with some DML,
This pacth fixes store/onlineBackupTest1.java failure on non-windows envirorment. Problem was unlogged operations thread and insert thread are working on the same connection. Test was failing becuase insert thread was committing the unlogged operation that was suppose to block the backup.
This pacth modified the test , so that these threads works on different conenctions,
This patch makes online backup call to wait/fail when unlogged operations like
create index are pending. It also fixes DERBY-523 bug my making the existing
log archive backup procedure to wait for the unlogged operation to complete.
-- Two new procedures are added to allow the users to make backup wait/fail
when unlogged operations are pending.
-- prevents users starting backup in an non-idle transactions to avoid backup
blocking forever if users starts backup in the same transaction as an
unlogged operation.
-- backup is not really transactional , to avoid any locking issues in the
futures; backup procedures ends the transaction by implicitly doing commit
when it is successful or rollback on any errors.
A new backup test is added to store suite to test the above scenarios.
DERBY-239, committing on behalf of Suresh Thalamati.
This patch adds code to support online backup when jar operations
are running parallel to the backup. Jar files are not logged, but the
system catalogs updates are logged when a jar file is added/replaced.
If the jar file operations are allowed during the backup, system catalog
(sys.sysfiles) table in the backup database can have a reference to a jar file
that does not exist in the backup database. And also backup can contain
partial written jar files. To make a consistent online backup, this patch:
1) Makes Backup operation wait/fail for all the jar operations activity in progress to complete.
2) Blocks jar file operations when a backup is in progress.
This patch also adds a new test to test the online backup
with jar operations.
DERBY-239, most recent patch committed on behalf of Suresh Thalamati
This patch addresses the issues raised by ·¹ystein in his review of previous
online backup patches 3-6.
- changed the backup procedures names with ONLINE to NOWAIT
- removed the transaction Idle restriction to run backup procedures.
- removed implicit commit/rollbacks.
- Added a new lesser impact restriction, which only disallows backup call only if
there are unlogged operations executed in the same transaction before the backup.
- Removed casting to RawTransaction.
- fixed Names and Comments.
- Enhanced the tests with addional test cases suggested by ·¹ystein.
DERBY-304 submitted on behalf of Suresh Thalamati
Fix :
1) Do not allow backup path to be any derby database directory. A directory is
assumed to be a derby database directory if it has service.properties file in it.
2) copy files needed from the database home into the backup one by one
instead of recursive copy from the top directory.
3) while copying the directories under jar directory, copy each sub directory
separately without copying any subdirectories under them (There should not
be any unless if user has created explicitly or created backup at that location).
4) Log and Seg0 directory are NOT already copied recursively, this was changed as part of
online backup work (DERBY-239).
Improvement
Closed
Major
Need a online backup feature that does not block update operations when online backup is in progress.
