[HBASE-16232] ITBLL fails on branch-1.3, now loosing actual keys

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.3.0, 1.2.1
• Component/s: dataloss, integration tests
• Assignee: Unassigned
• Reporter: Mikhail Antonov
• Created: 14/Jul/16 21:43
• Updated: 11/Jun/22 18:59
• Resolved: 11/Jun/22 18:59

Description

So I'm running ITBLL off branch-1.3 on recent commit (after stack's fix for fake keys showing up in the scans) with increased number of regions per regionserver and seeing the following.

$Verify​$Counts
REFERENCED 0 4,999,999,994 4,999,999,994
UNDEFINED 0 3 3
UNREFERENCED 0 3 3

So we're loosing some keys. This time those aren't fake:

undef
\x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3 0 1 1
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
unref
\x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
\xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15

Re-running verify step with CM off still shows this issue. Search tool reports:

Total
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3 5 0 5
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
CELL_WITH_MISSING_ROW 15 0 15

Will post more as I dig into.