Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9207

PeerSync recovery fails if number of updates requested is high

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 5.1, 6.0
    • 6.2, 7.0
    • None
    • None

    Description

      PeerSync recovery fails if we request more than ~99K updates.

      If update solrconfig to retain more tlogs to leverage https://issues.apache.org/jira/browse/SOLR-6359

      During out testing we found out that recovery using PeerSync fails if we ask for more than ~99K updates, with following error

       WARN  PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=<shardUrl>
      exception talking to <leaderUrl>, failed
      org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got application/xml. 
      <?xml version="1.0" encoding="UTF-8"?>
      <response>
      <lst name="error"><str name="msg">application/x-www-form-urlencoded content length (4761994 bytes) exceeds upload limit of 2048 KB</str><in
      t name="code">400</int></lst>
      </response>
      

      We arrived at ~99K with following match

      • max_version_number = Long.MAX_VALUE = 9223372036854775807
      • bytes per version number = 20 (on the wire as POST request sends version number as string)
      • additional bytes for separator ,
      • max_versions_in_single_request = 2MB/21 = ~99864

      I could think of 2 ways to fix it
      1. Ask for about updates in chunks of 90K inside PeerSync.requestUpdates()

      2. Use application/octet-stream encoding

      Attachments

        1. SOLR-9207.patch
          17 kB
          Shalin Shekhar Mangar
        2. SOLR-9207.patch
          17 kB
          Pushkar Raste
        3. SOLR-9207.patch_updated
          16 kB
          Pushkar Raste

        Issue Links

          Activity

            People

              shalin Shalin Shekhar Mangar
              praste Pushkar Raste
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: