[HBASE-21745] Make HBCK2 be able to fix issues other than region assignment

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
• Component/s: hbase-operator-tools, hbck2
• Hadoop Flags: Reviewed

Release Note

This issue adds via its subtasks:

 * An 'HBCK Report' page to the Master UI added by HBASE-22527+HBASE-22709+HBASE-22723+ (since 2.1.6, 2.2.1, 2.3.0). Lists consistency or anomalies found via new hbase:meta consistency checking extensions added to CatalogJanitor (holes, overlaps, bad servers) and by a new 'HBCK chore' that runs at a lesser periodicity that will note filesystem orphans and overlaps as well as the following conditions:
 ** Master thought this region opened, but no regionserver reported it.
 ** Master thought this region opened on Server1, but regionserver reported Server2
 ** More than one regionservers reported opened this region
 Both chores can be triggered from the shell to regenerate ‘new’ reports.
 * Means of scheduling a ServerCrashProcedure (HBASE-21393).
 * An ‘offline’ hbase:meta rebuild (HBASE-22680).
 * Offline replace of hbase.version and hbase.id
 * Documentation on how to use completebulkload tool to ‘adopt’ orphaned data found by new HBCK2 ‘filesystem’ check (see below) and ‘HBCK chore’ (HBASE-22859)
 * A ‘holes’ and ‘overlaps’ fix that runs in the master that uses new bulk-merge facility to collapse many overlaps in the one go.
 * hbase-operator-tools HBCK2 client tool got a bunch of additions:
 ** A specialized 'fix' for the case where operators ran old hbck 'offlinemeta' repair and destroyed their hbase:meta; it ties together holes in meta with orphaned data in the fs (HBASE-22567)
 ** A ‘filesystem’ command that reports on orphan data as well as bad references and hlinks with a ‘fix’ for the latter two options (based on hbck1 facility updated).
 ** Adds back the ‘replication’ fix facility from hbck1 (HBASE-22717)

The compound result is that hbck2 is now in excess of hbck1 abilities. The provided functionality is disaggregated as per the hbck2 philosophy of providing 'plumbing' rather than 'porcelain' so there is work to do still adding fix-it playbooks, scripting across outages, and automation.

• Assignee: Michael Stack
• Reporter: Duo Zhang
• Created: 19/Jan/19 01:56
• Updated: 22/Jun/22 00:09
• Resolved: 11/Sep/19 01:14

Description

This is what apurtell posted on mailing-list, HBCK2 should support

• Rebuild meta from region metadata in the filesystem, aka offline meta rebuild.
• Fix assignment errors (undeployed regions, double assignments (yes, should not be possible), etc) (See https://issues.apache.org/jira/browse/HBASE-21745?focusedCommentId=16888302&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16888302)
• Fix region holes, overlaps, and other errors in the region chain (See HBASE-22796 and HBASE-22771 – adds hole and overlap fixing to master; hbck2 client can as for a fixMeta).
• Fix failed split and merge transactions that have failed to roll back due to some bug (related to previous) (Previous items 'overlaps' will take care of these).
• Enumerate store files to determine file level corruption and sideline corrupt files
• Fix hfile link problems (dangling / broken)

Issue Links

• is related to: HBASE-25159 [hbck2] Add an 'adoption service' for 'Orphaned Regions'
• is related to: HBASE-22567 [HBCK2] Add new methods for dealing with missing regions in META while Master is online
• relates to: HBASE-21665 OfflineMetaRepair tool fails with NPE
• relates to: HBASE-27147 [HBCK2] extraRegionsInMeta does not work If RegionInfo is null
• relates to: HBASE-21447 HBCK2 tool have questions on holes when HBCK2 checks region chain
• relates to: HBASE-22707 [HBCK2] MasterRpcServices assigns method should try to reload regions from meta if the passed regions isn't found under AssignmentManager RegionsStateStore
• relates to: HBASE-22726 [HBCK2] Changes on HBASE-21585 breaks compatibility of HBCK2 with hbase version 3.0.0
• relates to: HBASE-22977 [HBCK2] hbck2 2.0
• relates to: HBASE-19121 HBCK for AMv2 (A.K.A HBCK2)
• links to: hbck2 additions

Sub-Tasks

• [hbck2] Add a master web ui to show the problematic regions: Resolved
• Fix failed split and merge transactions that have failed to roll back: Resolved
• Fix region holes, overlaps, and other region related errors: Resolved
• Add an API ScheduleSCP() to HBCK2: Resolved
• [HBCK2] OfflineMetaRepair for hbase2/hbck2: Resolved
• Add to migration doc that meta should be healthy before upgrade: Resolved
• [HBCK2] Add filesystem fixup to hbck2: Resolved
• HBCK - Add offline create/fix hbase.version and hbase.id: Closed
• Avoid to expose protobuf stuff in Hbck interface: Resolved
• Add a chore thread in master to do hbck checking and display results in 'HBCK Report' page: Resolved
• [HBCK2] Add hdfs integrity report to 'filesystem' command: Resolved
• [HBCK2] Expose replication fixes from hbck1: Resolved
• Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta: Resolved
• Add a new admin method and shell cmd to trigger the hbck chore to run: Resolved
• Show catalogjanitor consistency complaints in new 'HBCK Report' page: Resolved
• [HBCK2] Add more log for hbck operations at master side: Resolved
• [HBCK2] fixMeta method and server-side support: Resolved
• Add a multi-region merge (for fixing overlaps, etc.): Resolved
• [HBCK2] Add fix of overlaps to fixMeta hbck Service: Resolved
• Modify config value range to enable turning off of the hbck chore: Resolved
• Fix broken unit test, TestCatalogJanitorCluster on branch-2.1 and branch-2.0: Resolved
• HBCK Report showed wrong orphans regions on FileSystem: Resolved
• HBCK Report showed the offline regions which belong to disabled table: Resolved
• Show filesystem path for the orphans regions on filesystem: Resolved
• [HBCK2] Add a client-side to hbase-operator-tools that can exploit fixMeta added in server side: Resolved
• [HBCK2] Fix HBCK2 after HBASE-22777 & HBASE-22758: Resolved
• Revert MetaTableAccessor#makePutFromTableState access to public: Resolved
• Add HBCK Report to master's header.jsp: Resolved
• [HBCK2] Fix the orphan regions on filesystem: Resolved
• [HBCK2] shows the whole help/usage message after the error message: Closed
• [HBCK2] reference file check fails if compiled with old version but check against new: Resolved
• Move to SLF4J: Resolved
• Fix NOTICE and LICENSE: Resolved
• Should not show split parent regions in hbck report UI: Resolved
• HBCK report UI showed -1 if hbck chore not running: Resolved