Lucene - Core
  1. Lucene - Core
  2. LUCENE-2500

A Linux-specific Directory impl that bypasses the buffer cache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: modules/other
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I've been testing how we could prevent Lucene's merges from evicting
      pages from the OS's buffer cache. I tried fadvise/madvise (via JNI)
      but (frustratingly), I could not get them to work (details at
      http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html).

      The only thing that worked was to use Linux's O_DIRECT flag, which
      forces all IO to bypass the buffer cache entirely... so I created a
      Linux-specific Directory impl to do this.

      1. LUCENE-2500.patch
        25 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Attached patch, with new Directory impl DirectIOLinuxDirectory.

        Show
        Michael McCandless added a comment - Attached patch, with new Directory impl DirectIOLinuxDirectory.
        Hide
        Michael McCandless added a comment -

        Uwe found this nice ant plugin: http://ant-contrib.sourceforge.net/cpptasks/index.html

        It would allow us to [optionally] build the dynamic lib (JNI extension) from ant.

        But I'm not going to have any time any time soon to try it... and I'm not good with The Ant.

        Show
        Michael McCandless added a comment - Uwe found this nice ant plugin: http://ant-contrib.sourceforge.net/cpptasks/index.html It would allow us to [optionally] build the dynamic lib (JNI extension) from ant. But I'm not going to have any time any time soon to try it... and I'm not good with The Ant.
        Hide
        Jason Rutherglen added a comment -

        DirectIOLinuxDirectory is in trunk and works? Are we using it with segment merging yet? Perhaps a separate Jira issue?

        Show
        Jason Rutherglen added a comment - DirectIOLinuxDirectory is in trunk and works? Are we using it with segment merging yet? Perhaps a separate Jira issue?
        Hide
        Michael McCandless added a comment -

        It does work, but, it's not "general purpose", ie it's up to the app to only use it for indexing and not searching.

        With IOContext we should be able to make it general purpose, ie whether to use direct IO or not can be determined by looking at the context.

        Show
        Michael McCandless added a comment - It does work, but, it's not "general purpose", ie it's up to the app to only use it for indexing and not searching. With IOContext we should be able to make it general purpose, ie whether to use direct IO or not can be determined by looking at the context.
        Hide
        Christian Kohlschütter added a comment -

        I guess it would not be difficult to add Mac OS X support (via F_NOCACHE)?

        see http://evanjones.ca/write-latency-alignment.html

        Show
        Christian Kohlschütter added a comment - I guess it would not be difficult to add Mac OS X support (via F_NOCACHE)? see http://evanjones.ca/write-latency-alignment.html
        Hide
        Michael McCandless added a comment -

        Nice!

        Actually I'd like to generalize this Dir impl to be a UnixFSDirectory (adding ifdefs to handle the flags for the various flavors), and, fix it, once we have IOContext, to properly decide when to use direct IO and when not to. This way it's safe to just use on any Unix platform... (see LUCENE-2795).

        Show
        Michael McCandless added a comment - Nice! Actually I'd like to generalize this Dir impl to be a UnixFSDirectory (adding ifdefs to handle the flags for the various flavors), and, fix it, once we have IOContext, to properly decide when to use direct IO and when not to. This way it's safe to just use on any Unix platform... (see LUCENE-2795 ).
        Hide
        Grant Ingersoll added a comment -

        Bulk close for 3.1

        Show
        Grant Ingersoll added a comment - Bulk close for 3.1

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development