[HBASE-21843] RegionGroupingProvider breaks the meta wal file name pattern which may cause data loss for meta region

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.1.0, 2.2.0
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.3, 2.0.5, 2.3.0
• Component/s: wal
• Labels: data-loss
• Hadoop Flags: Reviewed
• Assignee: Wellington Chevreuil
• Reporter: Wellington Chevreuil
• Created: 04/Feb/19 15:29
• Updated: 28/Aug/19 05:50
• Resolved: 07/Feb/19 00:58

Description

A bit unusual, but managed to face this twice lately on both distributed and local standalone mode, on VMs. Somehow, after some VM pause/resume, got into a situation where regions on meta were assigned to a give RS startcode that had no corresponding WAL dir.

That caused those regions to never get assigned, because the given RS startcode is not found anywhere by RegionServerTracker/ServerManager, so no SCP is created to this RS startcode, leaving the region "open" on a dead server forever, in META.

Could get this sorted by adding extra check on loadMeta, checking if the RS assigned to the region in meta is not online and doesn't have a WAL dir, then mark this region as offline. 

Issue Links

• duplicates: HBASE-21519 Namespace region is never assigned in a HM failover scenario when multiwal is enabled
• is broken by: HBASE-20856 PITA having to set WAL provider in two places
• is broken by: HBASE-14448 Refine RegionGroupingProvider Phase-2: remove provider nesting and formalize wal group name