[HBASE-23143] Region Server Crash due to 2 cells out of order ( between 2 DELETEs)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.3.2
• Assignee: Unassigned
• Reporter: Xu Cang
• Created: 09/Oct/19 22:50
• Updated: 11/Jun/22 19:19
• Resolved: 11/Jun/22 19:19

Description

Region Server Crash due to 2 cells out of order ( between 2 DELETEs)

 

Caused by: java.io.IOException: Added a key not lexically larger than previous.
Current cell = 00D7F000000xxQ10D52v00008UY6yV0057F000000bPaGT\x000000000000000000057F000000bPaG/0:TABLE1_ID/1570095189597/DeleteColumn/vlen=0/seqid=2128373,
lastCell = 00D7F000000xxQ10D52v00008UY6yV0057F000000bPaGT\x000000000000000000057F000000bPaG/0:TABLE1_ID/1570095165147/DeleteColumn/vlen=0/seqid=2128378

 

 

I am aware of this JIRA: https://issues.apache.org/jira/browse/HBASE-22862

Though it's slightly different, HBASE-22862 issue was caused One Delete and One Put.

This issue I am reporting is caused by 2 Deletes

 

Has anyone seen this issue? 

 

After I read the code and debugged the test cases.

In AbstractHFileWriter.java

int keyComp = comparator.compareOnlyKeyPortion(lastCell, cell);

This call will always ignore SequenceId. And time stamps are in the correct order (above case)

And since these 2 cells have same KEY. The comparison result should be 0.
only possible issue I can think of is, in this code piece: in CellComparator.java:

Bytes.compareTo(left.getRowArray(), left.getRowOffset(), left.getRowLength(),
 right.getRowArray(), right.getRowOffset(), right.getRowLength());

The getRowLength() returns a wrong value.

Or the offset is messed up.