[SOLR-9207] PeerSync recovery fails if number of updates requested is high

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.1, 6.0
• Fix Version/s: 6.2, 7.0
• Assignee: Shalin Shekhar Mangar
• Reporter: Pushkar Raste
• Created: 13/Jun/16 15:35
• Updated: 06/Oct/21 16:36
• Resolved: 06/Jul/16 21:06

Description

PeerSync recovery fails if we request more than ~99K updates.

If update solrconfig to retain more tlogs to leverage https://issues.apache.org/jira/browse/SOLR-6359

During out testing we found out that recovery using PeerSync fails if we ask for more than ~99K updates, with following error

 WARN  PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
exception talking to <leaderUrl>, failed
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml. 
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
t name="code">400</int></lst>
</response>

We arrived at ~99K with following match

• max_version_number = Long.MAX_VALUE = 9223372036854775807
• bytes per version number = 20 (on the wire as POST request sends version number as string)
• additional bytes for separator ,
• max_versions_in_single_request = 2MB/21 = ~99864

I could think of 2 ways to fix it
1. Ask for about updates in chunks of 90K inside PeerSync.requestUpdates()

2. Use application/octet-stream encoding

Issue Links

• causes: SOLR-14058 IOOBE in PeerSync
• relates to: SOLR-11475 Endless loop and OOM in PeerSync