[HBASE-22378] HBase Canary fails with TableNotFoundException when table deleted during Canary run

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 1.3.0, 1.4.0, 1.5.0, 2.0.0, 2.1.5, 2.2.1
• Fix Version/s: 3.0.0-alpha-1, 1.5.0, 1.4.10, 2.3.0, 2.1.5, 2.2.1, 1.3.5
• Component/s: canary
• Labels: Canary
• Assignee: Caroline Zhou
• Reporter: Caroline Zhou
• Created: 08/May/19 00:33
• Updated: 23/Jan/20 14:10
• Resolved: 13/May/19 19:47

Description

In 1.3.2 branch-1, we saw a drastic increase in TableNotFoundExceptions thrown by HBase Canary. We traced the issue back to Canary trying to call isTableEnabled() on temporary tables that were deleted in the middle of the Canary run.

In this version of HBase Canary, Canary throws TableNotFoundException (and then fails) if a table is deleted between admin.listTables() and admin.tableEnabled() function calls in RegionMonitor's sniff() method. Following the goal of RegionMonitor.sniff(), which is to query all existing tables, in order to reduce noise we should skip over a table (i.e. don't check if it was enabled, or do anything else with it at all) if it was returned in listTables() but deleted before Canary can query it. Temporary tables which are not meant to be kept should not throw TableNotFoundExceptions which fail the Canary.

Patch in progress:

Add a call to admin.tableExists() before tableEnabled() on line 1244 in RegionMonitor.sniff().