Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/store
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.

      I have been benchmarking queries sequentially and was pretty surprised at how much faster
      MMapDirectory is, for example for cases that do many seeks.

      I think we should change the defaults for windows as such:

      if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
      use MMapDirectory
      else
      use SimpleFSDirectory

      I think we should just consider doing this for 4.0 only and see how it goes.

      1. LUCENE-2650.patch
        5 kB
        Robert Muir
      2. LUCENE-2650.patch
        5 kB
        Robert Muir

        Issue Links

          Activity

          Hide
          Robert Muir added a comment -

          Here's the patch:

          • sets the defaults as described in the issue
          • turns on unmap by default if its possible
          • when unmapping inputs, we should be careful to never allow the region to be accessed.
            so in close(), we ensure that clones, too set their buffer to null.
            and in clone() we add a friendlier exception rather than NPE.
          Show
          Robert Muir added a comment - Here's the patch: sets the defaults as described in the issue turns on unmap by default if its possible when unmapping inputs, we should be careful to never allow the region to be accessed. so in close(), we ensure that clones, too set their buffer to null. and in clone() we add a friendlier exception rather than NPE.
          Hide
          Robert Muir added a comment -

          by the way, all lucene/solr tests pass on 64-bit windows with -Dtests.directory=FSDirectory

          Show
          Robert Muir added a comment - by the way, all lucene/solr tests pass on 64-bit windows with -Dtests.directory=FSDirectory
          Hide
          Yonik Seeley added a comment -

          I think we should just consider doing this for 4.0 only and see how it goes.

          +1, that does seem to be the best option for windows servers at this point.

          Show
          Yonik Seeley added a comment - I think we should just consider doing this for 4.0 only and see how it goes. +1, that does seem to be the best option for windows servers at this point.
          Hide
          Robert Muir added a comment -

          same patch, only i added the same safety (ensuring clones null their buffers) to the MultiMMapII that I added to the MMapII.

          Show
          Robert Muir added a comment - same patch, only i added the same safety (ensuring clones null their buffers) to the MultiMMapII that I added to the MMapII.
          Hide
          Robert Muir added a comment -

          by the way, i ran all tests with -Dtests.directory=MMapDirectory under both Oracle/BEA Jrockit and IBM (although this one: 32 bit)
          both these support the unmap hack and all lucene/solr tests pass (except for an unrelated thai issue: LUCENE-2653)

          so i'd like to commit this to trunk only in a few days if no one objects.

          Show
          Robert Muir added a comment - by the way, i ran all tests with -Dtests.directory=MMapDirectory under both Oracle/BEA Jrockit and IBM (although this one: 32 bit) both these support the unmap hack and all lucene/solr tests pass (except for an unrelated thai issue: LUCENE-2653 ) so i'd like to commit this to trunk only in a few days if no one objects.
          Hide
          Robert Muir added a comment -

          I'm going to add the extra safety here for cloned mmapindexinputs as a separate commit
          from changing the defaults (in case we have to revert the defaults).

          Its also good to backport (unlike the defaults)

          Show
          Robert Muir added a comment - I'm going to add the extra safety here for cloned mmapindexinputs as a separate commit from changing the defaults (in case we have to revert the defaults). Its also good to backport (unlike the defaults)
          Hide
          Robert Muir added a comment -

          Committed revision 999409 to trunk.

          Show
          Robert Muir added a comment - Committed revision 999409 to trunk.
          Hide
          Robert Muir added a comment -

          I'd like to reopen this issue to propose backporting the defaults change (for 64-bit windows) to 3.x

          Its been a few months now with this default in trunk and there haven't been any reported problems.
          Additionally our tests are much better now (e.g. we test all directory impls).

          Show
          Robert Muir added a comment - I'd like to reopen this issue to propose backporting the defaults change (for 64-bit windows) to 3.x Its been a few months now with this default in trunk and there haven't been any reported problems. Additionally our tests are much better now (e.g. we test all directory impls).
          Hide
          Robert Muir added a comment -

          Committed revision 1043666 (branch_3x)

          Show
          Robert Muir added a comment - Committed revision 1043666 (branch_3x)
          Hide
          Grant Ingersoll added a comment -

          Bulk close for 3.1

          Show
          Grant Ingersoll added a comment - Bulk close for 3.1

            People

            • Assignee:
              Robert Muir
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development