[HBASE-28460] Full backup restore fails for empty HFiles

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-2
• Component/s: backup&restore
• Labels: pull-request-available
• Assignee: Dieter De Paepe
• Reporter: Dieter De Paepe
• Created: 27/Mar/24 15:00
• Updated: 03/Apr/24 20:25
• Resolved: 02/Apr/24 13:14

Description

A full backup restore fails if the backup contains an empty HFile, for example when all data has been deleted from a table and full compaction has run. There are several issues:

• HFiles are read in `RestoreTool` to read the first/last key, but this fails for empty HFiles
• In `RestoreTool`, table creation also incorrectly assumes the region contains keys
• In `MapReduceRestoreJob`, the tool incorrectly assumes that a bulkload with no loaded entries is an error.

Example stacktrace:

24/03/21 18:38:09 ERROR org.apache.hadoop.hbase.backup.util.BackupUtils: java.util.NoSuchElementException: No value present
java.util.NoSuchElementException: No value present
	at java.base/java.util.Optional.get(Optional.java:143)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.generateBoundaryKeys(RestoreTool.java:440)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.checkAndCreateTable(RestoreTool.java:493)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.createAndRestoreTable(RestoreTool.java:351)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.fullRestoreTable(RestoreTool.java:211)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:151)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:229)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.execute(RestoreTablesClient.java:265)
	at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.restore(BackupAdminImpl.java:518)
	at org.apache.hadoop.hbase.backup.RestoreDriver.parseAndRun(RestoreDriver.java:176)
	at org.apache.hadoop.hbase.backup.RestoreDriver.doWork(RestoreDriver.java:216)
	at org.apache.hadoop.hbase.backup.RestoreDriver.run(RestoreDriver.java:252)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
	at org.apache.hadoop.hbase.backup.RestoreDriver.main(RestoreDriver.java:224)
24/03/21 18:38:09 ERROR org.apache.hadoop.hbase.backup.RestoreDriver: Error while running restore backup
java.lang.IllegalStateException: Cannot restore hbase table
	at org.apache.hadoop.hbase.backup.util.RestoreTool.createAndRestoreTable(RestoreTool.java:360)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.fullRestoreTable(RestoreTool.java:211)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:151)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:229)
	at org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.execute(RestoreTablesClient.java:265)
	at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.restore(BackupAdminImpl.java:518)
	at org.apache.hadoop.hbase.backup.RestoreDriver.parseAndRun(RestoreDriver.java:176)
	at org.apache.hadoop.hbase.backup.RestoreDriver.doWork(RestoreDriver.java:216)
	at org.apache.hadoop.hbase.backup.RestoreDriver.run(RestoreDriver.java:252)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
	at org.apache.hadoop.hbase.backup.RestoreDriver.main(RestoreDriver.java:224)
Caused by: java.util.NoSuchElementException: No value present
	at java.base/java.util.Optional.get(Optional.java:143)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.generateBoundaryKeys(RestoreTool.java:440)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.checkAndCreateTable(RestoreTool.java:493)
	at org.apache.hadoop.hbase.backup.util.RestoreTool.createAndRestoreTable(RestoreTool.java:351)
	... 10 more 

Issue Links

• links to: GitHub Pull Request #5782