[SOLR-9915] PeerSync alreadyInSync check is not backwards compatible

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 6.3
• Fix Version/s: 6.4
• Component/s: replication (java)
• Assignee: Noble Paul
• Reporter: Tim Owen
• Created: 03/Jan/17 16:15
• Updated: 08/Jun/19 15:29
• Resolved: 04/Jan/17 06:39

Description

The fingerprint check added to PeerSync in SOLR-9446 works fine when all servers are running 6.3 but this means it's hard to do a rolling upgrade from e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to get a fingerprint and then gets a NPE because the older server doesn't return the expected field in its response.

This leads to the PeerSync completely failing, and results in a full index replication from scratch, copying all index files over the network. We noticed this happening when we tried to do a rolling upgrade on one of our 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering our disks and network, so we had to do a full shutdown, upgrade all to 6.3 and restart, which was not ideal for a production cluster.

The attached patch should behave more gracefully in this situation, as it will typically return false for alreadyInSync() and then carry on doing the normal re-sync based on versions.