[HBASE-18247] Hbck to fix the case that replica region shows as key in the meta table

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha-1
• Fix Version/s: 1.4.0, 1.3.2, 1.2.7
• Component/s: hbck, Operability
• Release Note: The hbck tool can no correct the meta table should it get an entry for a read replica region.
• Assignee: Hua Xiang
• Reporter: Hua Xiang
• Created: 21/Jun/17 00:26
• Updated: 01/Aug/18 06:23
• Resolved: 22/Oct/17 05:35

Description

Recently, we run into one case with read replica, the replica region shows up as key in meta table (it is not supposed to happen, we are still working on why it showed up in the meta table).

However, hbck always reported the error about the primary region. Please see the error attached.

The entry in meta table
test,92b0201b,1492546349354_0001.c3e6f235fe7caef75f8b0fb92a012da3. column=info:regioninfo, timestamp=1494958820573, value={ENCODED => c3e6f235fe7caef75f8b0fb92a012da3, NAME => 'test,92b0201b,1492546349354_0001.c3e6f235fe7caef75f8b0fb92a012da3.', STARTKEY => '92b0201b', ENDKEY => '92f1a952', REPLICA_ID => 1}

ERROR: Region { meta => test,92b0201b,1492546349354.d2c637715f31a072f174e70d407fb458., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.

Traced the code, in the following line, it does not consider the case that replicaId in regionInfo could be non-default.

https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java#L985

If it is changed to get replicaId from regionInfo, then hbck should be able to fix this by "-fixMeta".

diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
index 9eb5111..1649e53 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
@@ -982,7 +982,7 @@ public class MetaTableAccessor {
     List<HRegionLocation> locations = new ArrayList<>(1);
     NavigableMap<byte[],NavigableMap<byte[],byte[]>> familyMap = r.getNoVersionMap();
 
-    locations.add(getRegionLocation(r, regionInfo, 0));
+    locations.add(getRegionLocation(r, regionInfo, regionInfo.getReplicaId()));
 
     NavigableMap<byte[], byte[]> infoMap = familyMap.get(getCatalogFamily());
     if (infoMap == null) return new RegionLocations(locations);

Issue Links

• incorporates: HBASE-18362 hbck should not report split replica parent region from meta as errors
• Is contained by: HBASE-18223 Track the effort to improve/bug fix read replica feature
• is related to: HBASE-19063 Ensure branch-2 and master have needed equivalent to HBASE-18247 'Hbck to fix the case that replica region shows as key in the meta table'