[HBASE-28829] Increment Inconsistency in Replication

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Replication

Environment

OS: macOS Sonoma 14.6.1

Language

Java

• Assignee: Unassigned
• Reporter: Jeevan Prakash
• Created: 10/Sep/24 07:18
• Updated: 10/Sep/24 14:57

Description

Issue:
Consistency is not achieved for Increment operation in replication.

Setup:
Lets have two HBase clusters 'cluster1' and 'cluster2' and both are added as peers to each other in them and both have replication enabled. There is a counter cell with initial value '2' in a table. There is a replication delay from 'cluster1' to 'cluster2'.

Actions:
1. Perform increment in 'cluster1' with 1.
2. Perform increment in 'cluster2' with 2.

Expected Behaviour:
The value in the counter cell should be 5.

Actual Behaviour:
The value in the counter cell is 4.

Analysis:
1. After increment in 'cluster1', the value became 2 in 'cluster1'.
2. The replication from 'cluster1' to 'cluster2' gets initiated.
3. But there is a replication delay from 'cluster1' to 'cluster2' and within that timeframe, increment in 'cluster2' performed.
4. Now the value is 4 in 'cluster2' and it got replicated to 'cluster1'.
5. Since the replication is cell-based, not operation based and the 'cluster2' increment is the latest, value 4 from 'cluster2' overrides value 3 in 'cluster1'.

Steps to reproduce:
Add a coprocessor in 'cluster2' with 'Thread.sleep' in 'preWALAppend' to simulate replication delay.

Inference:
During debugging, it was discovered that the replication is cell-based, meaning the entire cell is being replicated rather than the specific operation being performed. This method works for other operations such as put and delete operations because it resolves the inconsistency problem by utilising the timestamp and version of the cell. However, for increment operations, which rely on the cell's previous value, this method is not successful.