[HBASE-23369] Auto-close 'unknown' Regions reported as OPEN on RegionServers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Hadoop Flags: Incompatible change
• Release Note: If a RegionServer reports a Region as OPEN in disagreement with Master's status on the Region, the Master now tells the RegionServer to silently close the Region.
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 05/Dec/19 06:27
• Updated: 03/Nov/20 05:48
• Resolved: 03/Jan/20 18:32

Description

In old days, if a RegionServer reported a variance that didn't agree w/ Master view of the cluster, we'd kill the RegionServer.

Lately, in tests that overrun a cluster, after a sustained high-load, Master can start failing its updates against Meta (CallQueueTooBigException <= More on this later). It then can lose proper accounting of all Region members. One variant has a RegionServer reporting its list of open Regions to the Master and the Master doesn't 'know' of a particular Region or the Master may know the Region but expects it open on another RegionServer.

Here is an example of how it looks each time RS reports:

 2019-12-03 07:07:00,757 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: No t1,08f5c285,1573094375485.ee78a0c951c1c902d8f3f3912394a0e5. RegionStateNode but reported ONLINE at server.example.org,16020,1575354666245 (inServerRegionList=false).
 2019-12-03 07:07:03,793 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: No t1,08f5c285,1573094375485.ee78a0c951c1c902d8f3f3912394a0e5. RegionStateNode but reported ONLINE at server.example.org,16020,1575354666245 (inServerRegionList=false).

Will also show as an 'inconsistency' in the 'HBCK' tab on the Master UI.

Issue Links

• links to: GitHub Pull Request #902