Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13029

Allow HDFS backup/restore buffer size to be configured

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.5, 8.0
    • Fix Version/s: 7.7, 8.0, master (9.0)
    • Component/s: Backup/Restore, hdfs
    • Labels:
      None

      Description

      There's a default hardcoded buffer size setting of 4096 in the HDFS code which means in particular that restoring a backup from HDFS takes a long time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes is very inefficient. We changed this in our local build used in production to 256kB and saw a 10x speed improvement when restoring a backup. Attached patch simply makes this size configurable using a command line setting, much like several other buffer size values.

        Attachments

        1. SOLR-13029.patch
          7 kB
          Tim Owen
        2. SOLR-13029.patch
          2 kB
          Tim Owen
        3. SOLR-13029.patch
          0.8 kB
          Tim Owen

          Activity

            People

            • Assignee:
              mkhl Mikhail Khludnev
              Reporter:
              TimOwen Tim Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: