Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8438

RAMDirectory speed improvements and cleanup

Details

    • Improvement
    • Status: Reopened
    • Minor
    • Resolution: Fixed
    • None
    • 9.0
    • None
    • None
    • New

    Description

      RAMDirectory screams for a cleanup. It is used and abused in many places and even if we discourage its use in favor of native (mmapped) buffers, there seem to be benefits of keeping RAMDirectory available (quick throw-away indexes without the need to setup external tmpfs, for example).

      Currently RAMDirectory performs very poorly under concurrent loads. The implementation is also open for all sorts of abuses – the streams can be reset and are used all around the place as temporary buffers, even without the presence of RAMDirectory itself. This complicates the implementation and is pretty confusing.

      An example of how dramatically slow RAMDirectory is under concurrent load, consider this PoC pseudo-benchmark. It creates a single monolithic segment with 500K very short documents (single field, with norms). The index is ~60MB once created. We then run semi-complex Boolean queries on top of that index from N concurrent threads. The attached capture-4 shows the result (queries per second over 5-second spans) for a varying number of concurrent threads on an AWS machine with 32 CPUs available (of which it seems 16 seem to be real, 16 hyper-threaded). That red line at the bottom (which drops compared to a single-threaded performance) is the current RAMDirectory. RAMDirectory2 is an alternative implementation I wrote that uses ByteBuffers. Yes, it's slower than the native mmapped implementation, but a lot faster then the current RAMDirectory (and more GC-friendly because it uses dynamic progressive block scaling internally).

      Attachments

        1. capture-1.png
          118 kB
          Dawid Weiss
        2. capture-4.png
          58 kB
          Dawid Weiss

        Issue Links

          Activity

            People

              dweiss Dawid Weiss
              dweiss Dawid Weiss
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m