[HBASE-26913] Replication Observability Framework

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-alpha-4
• Component/s: regionserver, Replication
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 01/Apr/22 13:59
• Updated: 20/Apr/23 21:34
• Resolved: 20/Apr/23 21:34

Description

In our production clusters, we have seen cases where data is present in source cluster but not in the sink cluster and 1 case where data is present in sink cluster but not in source cluster. 

We have internal tools where we take incremental backup every day on both source and sink clusters and we compare the hash of the data in both the backups. We have seen many cases where hash doesn't match which means data is not consistent between source and sink for that given day. The Mean Time To Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of manual debugging.

We need some tool where we can reduce MTTD and requires less manual debugging.

I have attached design doc. Huge thanks to bharathv  to come up with this design at my work place.

Issue Links

• relates to: HBASE-27461 Add multiWAL support for Replication Observability framework.
• links to: Design Doc
• links to: Github feature branch
• links to: GitHub Pull Request #4556
• links to: GitHub Pull Request #4862

Sub-Tasks

• Create WAL event tracker table to track all the WAL events.: Resolved
• Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.: Resolved
• Add documentation for Replication Observability Framework in hbase book.: Resolved