[HBASE-14987] Compaction marker whose region name doesn't match current region's needs to be handled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 0.98.17, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 16/Dec/15 00:25
• Updated: 08/Feb/16 04:57
• Resolved: 04/Jan/16 15:21

Description

One customer encountered the following error when replaying recovered edits, leading to region open failure:

region=table1,d6b-2282-9223370590058224807-U-9856557-        EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d., starting to roll back the global memstore   size.
org.apache.hadoop.hbase.regionserver.WrongRegionException: Compaction marker from WAL table_name: "table1"
encoded_region_name: "d389c70fde9ec07971d0cfd20ef8f575"
...
region_name: "table1,d6b-2282-9223370590058224807-U-9856557-EJ452727-16313786400171,1449089609367.d389c70fde9ec07971d0cfd20ef8f575."
 targetted for region d389c70fde9ec07971d0cfd20ef8f575 does not match this region: {ENCODED => fa8a526f2578eb3630bb08a4b1648f5d, NAME => 'table1,d6b-2282-                        9223370590058224807-U-9856557-EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.', STARTKEY => 'd6b-2282-9223370590058224807-U-9856557-EJ452727-             16313786400171', ENDKEY => 'd76-2553-9223370588576178807-U-7416904-EK875822-17662180600000'}
  at org.apache.hadoop.hbase.regionserver.HRegion.checkTargetRegion(HRegion.java:4592)
  at org.apache.hadoop.hbase.regionserver.HRegion.replayWALCompactionMarker(HRegion.java:3831)
  at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3747)
  at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3601)
  at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:911)
  at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:789)
  at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:762)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5774)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5744)

This was likely caused by the following action of hbck:

15/12/08 18:11:34 INFO util.HBaseFsck: [hbasefsck-pool1-t37] Moving files from hdfs://Zealand/hbase/data/default/table1/d389c70fde9ec07971d0cfd20ef8f575/recovered.edits into     containing region hdfs://Zealand/hbase/data/default/table1/fa8a526f2578eb3630bb08a4b1648f5d/recovered.edits

The recovered.edits for d389c70fde9ec07971d0cfd20ef8f575 contained compaction marker which couldn't be replayed against fa8a526f2578eb3630bb08a4b1648f5d