[COUCHDB-2965] Race condition in replicator rescan logic

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Replication
• Assignee: Unassigned
• Reporter: Nick Vatamaniuc
• Created: 07/Mar/16 16:32
• Updated: 09/May/16 21:39
• Resolved: 09/May/16 21:38

Description

There is race condition between the full rescan and regular change feed processing in the couch_replicator_manger code.

This race condition would lead to replication docs left in untriggered state when a rescan of all the docs is performed. The rescan might happen when nodes connect and disconnect. The likelihood of this race condition appear goes up if a lot of documents are updated and there is a back-up of messages in the replicator manager's mailbox.

The race condition happens in the following way:

• A full rescan is initiated here: https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L424 It clears the db_to_seq ets table which holds the latest change sequence for each replicator database. Then launches a scan_all_dbs process.

• scan_all_dbs will find all replicator-looking-like database and for each send a {resume_scan, DbName} message to the main couch_replicator_manager process.

• {resume_scan, DbName} message is handled here: https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L233 The expectation is because db_to_seq was reset it ends up not finding a sequence checkpoint in db_to_seq, so start 0 and spawns a new change feed, which will rescan all documents (since we need to determine ownership for them).

But the race condition occurs because when change feeds stop, they call replicator manager with { rep_db_checkpoint, DbName } message. That updates db_to_seq ets table with the latest change sequence: https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L225 Which means this sequence of operations could happen:

• db_to_seq is reset to 0, scan_all_dbs is spawned

• change feed stops at sequence 1042, it calls {rep_db_checkpoint, <<"_replicator">>}

• {rep_db_checkpoint, <<"_replicator">>} call is handled, now latest db_to_seq for _replicator is 1042

• {resume, <<"_replicator">>} is sent from scan_all_dbs process and received by replicator manager. It sees that db_to_seq has _replicator with latest sequence 1042, so it will either start from that instead of 0, thus skipping updates from 0 to 1042.

This was seen by running the experiment with1000 replication documents were being updated. Around document 700 or so , node1 was killed (pkill -f node1) . node2 experienced the race condition on rescan and never picked up a bunch of document that should have belong to it. didn't.