[SOLR-13029] Allow HDFS backup/restore buffer size to be configured

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.5, 8.0
• Fix Version/s: 7.7, 8.0, master (9.0)
• Component/s: Backup/Restore, hdfs
• Assignee: Mikhail Khludnev
• Reporter: Tim Owen
• Created: 30/Nov/18 20:40
• Updated: 21/Nov/19 00:41
• Resolved: 25/Jan/19 07:45

Description

There's a default hardcoded buffer size setting of 4096 in the HDFS code which means in particular that restoring a backup from HDFS takes a long time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes is very inefficient. We changed this in our local build used in production to 256kB and saw a 10x speed improvement when restoring a backup. Attached patch simply makes this size configurable using a command line setting, much like several other buffer size values.