Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11707

allow to configure the HDFS block size

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Hadoop Integration, hdfs
    • Labels:
      None

      Description

      Currently index files are created in HDFS with the block size that is defined on the namenode. For that the HdfsFileWriter reads out the config from the server and then specifies the size (and replication factor) in the FileSystem.create call.

      For the write.lock files things work slightly different. These are being created by the HdfsLockFactory without specifying a block size (or replication factor). This results in a default being picked by the HDFS client, which is 128MB.

      So currently files are being created with different block sizes if the namenode is configured to something else then 128MB. It would be good if Solr would allow to configure the block size to be used. This is especially useful if the Solr admin is not the HDFS admin and if you have different applications using HDFS that have different requirements for their block size.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              HendrikH Hendrik Haddorp
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: