[SOLR-11707] allow to configure the HDFS block size

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Hadoop Integration, hdfs
• Assignee: Unassigned
• Reporter: Hendrik Haddorp
• Created: 30/Nov/17 08:35
• Updated: 08/Jun/19 15:12

Description

Currently index files are created in HDFS with the block size that is defined on the namenode. For that the HdfsFileWriter reads out the config from the server and then specifies the size (and replication factor) in the FileSystem.create call.

For the write.lock files things work slightly different. These are being created by the HdfsLockFactory without specifying a block size (or replication factor). This results in a default being picked by the HDFS client, which is 128MB.

So currently files are being created with different block sizes if the namenode is configured to something else then 128MB. It would be good if Solr would allow to configure the block size to be used. This is especially useful if the Solr admin is not the HDFS admin and if you have different applications using HDFS that have different requirements for their block size.