Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2373

Create a Codec to work with streaming and append-only filesystems

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Since early 2.x times Lucene used a skip/seek/write trick to patch the length of the terms dict into a place near the start of the output data file. This however made it impossible to use Lucene with append-only filesystems such as HDFS.

      In the post-flex trunk the following code in StandardTermsDictWriter initiates this:

          // Count indexed fields up front
          CodecUtil.writeHeader(out, CODEC_NAME, VERSION_CURRENT); 
      
          out.writeLong(0);                             // leave space for end index pointer
      

      and completes this in close():

            out.seek(CodecUtil.headerLength(CODEC_NAME));
            out.writeLong(dirStart);
      

      I propose to change this layout so that this pointer is stored simply at the end of the file. It's always 8 bytes long, and we known the final length of the file from Directory, so it's a single additional seek(length - 8) to read it, which is not much considering the benefits.

        Attachments

        1. appending.patch
          51 kB
          Andrzej Bialecki
        2. appending.patch
          48 kB
          Andrzej Bialecki
        3. LUCENE-2372-2.patch
          55 kB
          Andrzej Bialecki
        4. LUCENE-2373.patch
          53 kB
          Michael McCandless

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ab Andrzej Bialecki
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: