[HBASE-16458] Shorten backup / restore test execution time

• Type: Test
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 3.0.0-beta-2
• Labels: backup
• Hadoop Flags: Reviewed
• Assignee: Vladimir Rodionov
• Reporter: Ted Yu
• Created: 19/Aug/16 21:32
• Updated: 16/Dec/23 09:51

Description

Below was timing information for all the backup / restore tests (today's result):

Running org.apache.hadoop.hbase.backup.TestIncrementalBackup
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 576.273 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackup
Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.67 sec - in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.34 sec - in org.apache.hadoop.hbase.backup.TestBackupStatusProgress
Running org.apache.hadoop.hbase.backup.TestBackupAdmin
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.251 sec - in org.apache.hadoop.hbase.backup.TestBackupAdmin
Running org.apache.hadoop.hbase.backup.TestHFileArchiving
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.323 sec - in org.apache.hadoop.hbase.backup.TestHFileArchiving
Running org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.492 sec - in org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
Running org.apache.hadoop.hbase.backup.TestBackupDescribe
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.758 sec - in org.apache.hadoop.hbase.backup.TestBackupDescribe
Running org.apache.hadoop.hbase.backup.TestBackupLogCleaner
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.187 sec - in org.apache.hadoop.hbase.backup.TestBackupLogCleaner
Running org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 330.539 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
Running org.apache.hadoop.hbase.backup.TestRemoteBackup
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.371 sec - in org.apache.hadoop.hbase.backup.TestRemoteBackup
Running org.apache.hadoop.hbase.backup.TestBackupSystemTable
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.893 sec - in org.apache.hadoop.hbase.backup.TestBackupSystemTable
Running org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.779 sec - in org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
Running org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.815 sec - in org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
Running org.apache.hadoop.hbase.backup.TestBackupShowHistory
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.517 sec - in org.apache.hadoop.hbase.backup.TestBackupShowHistory
Running org.apache.hadoop.hbase.backup.TestRemoteRestore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.799 sec - in org.apache.hadoop.hbase.backup.TestRemoteRestore
Running org.apache.hadoop.hbase.backup.TestFullRestore
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 317.711 sec - in org.apache.hadoop.hbase.backup.TestFullRestore
Running org.apache.hadoop.hbase.backup.TestFullBackupSet
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.045 sec - in org.apache.hadoop.hbase.backup.TestFullBackupSet
Running org.apache.hadoop.hbase.backup.TestBackupDelete
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.214 sec - in org.apache.hadoop.hbase.backup.TestBackupDelete
Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.631 sec - in org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
Running org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 190.358 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
Running org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.726 sec - in org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination
Running org.apache.hadoop.hbase.backup.TestFullBackup
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 177.137 sec - in org.apache.hadoop.hbase.backup.TestFullBackup

The duration was close to 1 hour:

[INFO] Apache HBase - Server .............................. SUCCESS [  01:02 h]

We should:

• merge duplicate backup / restore test cases
• shorten backup / restore test execution time (ingesting less data, etc)
• use multi wal for selected test(s) since rolling / replaying more WAL files takes more time

Issue Links

• is part of: HBASE-17362 HBase Backup/Restore Phase 4
• relates to: HBASE-17949 Shorten the execution time of TestBackupMultipleDeletes