Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2373

Create a Codec to work with streaming and append-only filesystems

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0-ALPHA
    • core/index
    • None
    • New

    Description

      Since early 2.x times Lucene used a skip/seek/write trick to patch the length of the terms dict into a place near the start of the output data file. This however made it impossible to use Lucene with append-only filesystems such as HDFS.

      In the post-flex trunk the following code in StandardTermsDictWriter initiates this:

          // Count indexed fields up front
          CodecUtil.writeHeader(out, CODEC_NAME, VERSION_CURRENT); 
      
          out.writeLong(0);                             // leave space for end index pointer
      

      and completes this in close():

            out.seek(CodecUtil.headerLength(CODEC_NAME));
            out.writeLong(dirStart);
      

      I propose to change this layout so that this pointer is stored simply at the end of the file. It's always 8 bytes long, and we known the final length of the file from Directory, so it's a single additional seek(length - 8) to read it, which is not much considering the benefits.

      Attachments

        1. LUCENE-2373.patch
          53 kB
          Michael McCandless
        2. LUCENE-2372-2.patch
          55 kB
          Andrzej Bialecki
        3. appending.patch
          48 kB
          Andrzej Bialecki
        4. appending.patch
          51 kB
          Andrzej Bialecki

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ab Andrzej Bialecki
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: