Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9503

Fix comment explaining reason for 8KB buffered/chunked writes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 8.6.2
    • Fix Version/s: None
    • Component/s: core/store
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available
    • Review Patch?:
      Yes

      Description

      FSDirectory uses an 8KB buffer and chunk size when writing to disk. The class has a comment explaining how file channels would do mallocs for larger chunks. However, that comment was originally introduced when RandomAccessFile and later FileOutputStream was used. Since the switch to file channels, there is no longer any difference (except the size of the buffer) for different buffer/chunk sizes.

      Attached patch keeps the comment, clarifying that it is a historic reason.

      Including my research here for background information in case this helps validate the change: 

      1. The stack buffer for 8KB or below in the JDK is here:
        https://github.com/openjdk/jdk/blob/master/src/java.base/share/native/libjava/io_util.c#L150
        This is used by both FileOutputStream and RandomAccessFile.
      2. This was applicable when the comment to lucene was added and later changed to FileOutputStream:
        https://github.com/apache/lucene-solr/commit/c4bdb2dad0abd85427f36732f4b1be66787b9815#diff-dc399b5658d57753fffdad7539f22e50R344
      3. Lucene cut over to nio here:
        https://github.com/apache/lucene-solr/commit/3eb66fb19ca2aa3d9dce53661f3233b6c9d3f974#diff-dc399b5658d57753fffdad7539f22e50R324
        and the comment was later changed to say file channels here:
        https://github.com/apache/lucene-solr/commit/cf845b1335af05f27d746a38a043bd815821a69f
      4. File channels use thread local direct buffers instead, see sun.nio.ch.IOUtil.write, which uses Util.getTemporaryDirectBuffer to get a thread local buffer.

       

        Attachments

        1. LUCENE-8kb-comment.patch
          0.9 kB
          Henning Andersen

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              henningandersen Henning Andersen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: