Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1309

Using IndexSearcher in a multithreaded context causes contention

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.3.1, 2.3.2
    • None
    • None
    • None
      • WinXP, Dual Core Centrino
      • Opensuse Linux, Dual Core AMD

      java version "1.6.0_06"
      Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
      Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)

    • New

    Description

      The attached code allows to search with a single IndexSearcher across threads or with a single instance per Thread, additionally it allows to deals with String.intern in Field from LUCENE-1308.

      Really interesting is the behavior of the current implementation (Field String.intern) The performance improves dramatically.

      Is this by design? Because it is usually recommended to reuse the IndexSearcher. How about a version to supports mult-threading much better basically delivering a read-only view?

      The following results have been measured. The [main] part is a single threaded run. The following thread results

      Field String.intern, Single Searcher
      [main] Search took: 3453ms
      [Thread-2] Search took: 17812ms
      [Thread-3] Search took: 18313ms
      [Thread-1] Search took: 18234ms
      [Thread-0] Search took: 18562ms

      Field WeakHashMap, Single Searcher
      [main] Search took: 3156ms
      [Thread-3] Search took: 14953ms
      [Thread-1] Search took: 15593ms
      [Thread-0] Search took: 15656ms
      [Thread-2] Search took: 16188ms

      Field ConcurrentHashMap, Single Searcher
      [main] Search took: 2844ms
      [Thread-1] Search took: 14812ms
      [Thread-0] Search took: 14890ms
      [Thread-2] Search took: 15172ms
      [Thread-3] Search took: 14656ms

      Field intern, Searcher per Thread
      [main] Search took: 3687ms
      [Thread-2] Search took: 10766ms
      [Thread-1] Search took: 10734ms
      [Thread-0] Search took: 11047ms
      [Thread-3] Search took: 10938ms

      Field Weak, Searcher per Thread
      [main] Search took: 3187ms
      [Thread-3] Search took: 11625ms
      [Thread-2] Search took: 12484ms
      [Thread-1] Search took: 12609ms
      [Thread-0] Search took: 12391ms

      Field Concurrent, Searcher per Thread
      [main] Search took: 2938ms
      [Thread-3] Search took: 10203ms
      [Thread-2] Search took: 10360ms
      [Thread-1] Search took: 10406ms
      [Thread-0] Search took: 10719ms

      Attachments

        1. yad.zip
          1.85 MB
          Rene Schwietzke

        Issue Links

          Activity

            tomoko Tomoko Uchida added a comment -

            This issue was moved to GitHub issue: #2386.

            tomoko Tomoko Uchida added a comment - This issue was moved to GitHub issue: #2386 .

            Dup of LUCENE-1329 (readOnly IndexReader) and LUCENE-753 (NIOFSDirectory), at least on non-Windows platforms.

            Based on this user report it seems like sharing an IndexReader across multiple threads in fact gives better performance than thread-private IndexReader instances:

            http://www.nabble.com/IndexSearcher-and-multi-threaded-performance-td20449375.html

            mikemccand Michael McCandless added a comment - Dup of LUCENE-1329 (readOnly IndexReader) and LUCENE-753 (NIOFSDirectory), at least on non-Windows platforms. Based on this user report it seems like sharing an IndexReader across multiple threads in fact gives better performance than thread-private IndexReader instances: http://www.nabble.com/IndexSearcher-and-multi-threaded-performance-td20449375.html

            The above results were created running four threads.

            renes Rene Schwietzke added a comment - The above results were created running four threads.
            yseeley@gmail.com Yonik Seeley added a comment -

            How many threads were used to do this test?

            Because it is usually recommended to reuse the IndexSearcher

            Yes (actually the IndexReader is the important part to cache and re-use).
            Because of memory usage associated with an IndexReader, it would be much better to improve or fix multi-threaded behavior rather than use multiple IndexReaders.

            yseeley@gmail.com Yonik Seeley added a comment - How many threads were used to do this test? Because it is usually recommended to reuse the IndexSearcher Yes (actually the IndexReader is the important part to cache and re-use). Because of memory usage associated with an IndexReader, it would be much better to improve or fix multi-threaded behavior rather than use multiple IndexReaders.

            People

              Unassigned Unassigned
              renes Rene Schwietzke
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment