Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 7.5, 8.0
-
Fix Version/s: 7.7, 8.0, master (9.0)
-
Component/s: Backup/Restore, hdfs
-
Labels:None
Description
There's a default hardcoded buffer size setting of 4096 in the HDFS code which means in particular that restoring a backup from HDFS takes a long time. Copying multi-GB files from HDFS using a buffer as small as 4096 bytes is very inefficient. We changed this in our local build used in production to 256kB and saw a 10x speed improvement when restoring a backup. Attached patch simply makes this size configurable using a command line setting, much like several other buffer size values.