Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
Encountered on HBase a2e7d2015e9f603e46339d0582e29a86843b9324 (branch-2), running in Kubernetes.
Description
I was doing some experiments with HBase backup, and came to the following situation:
user@hadoop-client-769bc9946-g6p59:/$ hbase backup history
{ID=backup_1694682979468,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:16:20 UTC 2023,End time=Thu Sep 14 09:16:45 UTC 2023,Progress=100%}
{ID=backup_1694682942306,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:15:43 UTC 2023,End time=Thu Sep 14 09:16:11 UTC 2023,Progress=100%}
{ID=backup_1694682906267,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:15:08 UTC 2023,End time=Thu Sep 14 09:15:34 UTC 2023,Progress=100%}
{ID=backup_1694682869991,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 09:14:31 UTC 2023,End time=Thu Sep 14 09:14:58 UTC 2023,Progress=100%}
{ID=backup_1694681054166,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=FAILED,Start time=Thu Sep 14 08:44:16 UTC 2023,Failed message=REPAIR status: repaired after failure:
backup_1694681054166,Progress=0%}
{ID=backup_1694681006360,Type=FULL,Tables={foo:bar,foo:baz},State=COMPLETE,Start time=Thu Sep 14 08:43:27 UTC 2023,End time=Thu Sep 14 08:43:56 UTC 2023,Progress=100%}
Even though I was able to create new incremental backups, I am not able to merge the old backups, due to the failed (& repaired) backup:
# Failed incremental + completed incremental user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge backup_1694681054166,backup_1694682869991 23/09/14 09:17:57 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error running command-line tool java.io.IOException: Backup image backup_1694681054166 can not be merged becuase of its state: FAILED at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:664) at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619) at org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834) at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134) at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169) at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82) at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
# Completed full backup + failed incremental
user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge backup_1694681006360,backup_1694681054166
23/09/14 09:19:40 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error running command-line tool
java.io.IOException: FULL backup image can not be merged for:
backup_1694681006360
at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:659)
at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619)
at org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834)
at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134)
at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169)
at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
I'm not sure why merging a failed backup is not possible, given that it is possible to create new incremental backups on top of it. Either the data in that failed backup is reliable (meaning it should be merge-able), or not reliable (meaning it should not be possible to execute incremental backups on top of it).
If I were to delete the failed backup, all other incremental backups would also be deleted. This made me assume failed backups still keep track of reliable data.