Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13029

Allow HDFS backup/restore buffer size to be configured

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.5, 8.0
    • 7.7, 8.0, 9.0
    • Backup/Restore, hdfs
    • None

    Description

      There's a default hardcoded buffer size setting of 4096 in the HDFS code which means in particular that restoring a backup from HDFS takes a long time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes is very inefficient. We changed this in our local build used in production to 256kB and saw a 10x speed improvement when restoring a backup. Attached patch simply makes this size configurable using a command line setting, much like several other buffer size values.

      Attachments

        1. SOLR-13029.patch
          0.8 kB
          Tim Owen
        2. SOLR-13029.patch
          2 kB
          Tim Owen
        3. SOLR-13029.patch
          7 kB
          Tim Owen

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mkhl Mikhail Khludnev
            TimOwen Tim Owen
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment