Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9207

PeerSync recovery fails if number of updates requested is high

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 5.1, 6.0
    • Fix Version/s: 6.2, 7.0
    • Component/s: None
    • Labels:
      None

      Description

      PeerSync recovery fails if we request more than ~99K updates.

      If update solrconfig to retain more tlogs to leverage https://issues.apache.org/jira/browse/SOLR-6359

      During out testing we found out that recovery using PeerSync fails if we ask for more than ~99K updates, with following error

       WARN  PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
      exception talking to <leaderUrl>, failed
      org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml. 
      <?xml version="1.0" encoding="UTF-8"?>
      <response>
      <lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
      t name="code">400</int></lst>
      </response>
      

      We arrived at ~99K with following match

      • max_version_number = Long.MAX_VALUE = 9223372036854775807
      • bytes per version number = 20 (on the wire as POST request sends version number as string)
      • additional bytes for separator ,
      • max_versions_in_single_request = 2MB/21 = ~99864

      I could think of 2 ways to fix it
      1. Ask for about updates in chunks of 90K inside PeerSync.requestUpdates()

      2. Use application/octet-stream encoding

        Attachments

        1. SOLR-9207.patch
          17 kB
          Pushkar Raste
        2. SOLR-9207.patch_updated
          16 kB
          Pushkar Raste
        3. SOLR-9207.patch
          17 kB
          Shalin Shekhar Mangar

          Issue Links

            Activity

              People

              • Assignee:
                shalinmangar Shalin Shekhar Mangar
                Reporter:
                praste Pushkar Raste
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: