[HBASE-26874] VerifyReplication recompare async

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-1

Release Note

VerifyReplication can now do multiple recompares (rather than 1 previously), with exponential backoff. In order to speed up the job when there are many discrepancies, the recompares can be run in background threads. New options added: --recompareTries (default 1 if --recompareSleep is > 0, otherwise 0), --recompareThreads (default 0), --recompareBackoffExponent (default 1).

• Assignee: Hernan Gelaf-Romer
• Reporter: Bryan Beaudreault
• Created: 21/Mar/22 13:51
• Updated: 08/Aug/23 20:35
• Resolved: 01/Aug/23 14:31

Description

VerifyReplication includes an option "sleepMsBeforeReCompare". This is useful for helping work around replication lag. However, adding a sleep in a hadoop job can drastically slow that job down if there is anything more than a small number of invalid results.

We can mitigate this by doing the recompare in a separate thread. We can limit the thread pool and fallback to doing the recompare in the main thread if the thread pool is full. This way we offload some of the slowness but still retain the same validation guarantees. A configuration can be added to control how many threads per mapper.

Issue Links

• links to: GitHub Pull Request #5050
• links to: GitHub Pull Request #5051