Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3653

Lucene Search not scalling

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.6, 4.0-ALPHA
    • None
    • None
    • New, Patch Available

    Description

      I've noticed that when doing thousands of searches in a single thread the average time is quite low i.e. a few milliseconds. When adding more concurrent searches doing exactly the same search the average time increases drastically.
      I've profiled the search classes and found that the whole of lucene blocks on

      org.apache.lucene.index.SegmentCoreReaders.getTermsReader
      org.apache.lucene.util.VirtualMethod
      public synchronized int getImplementationDistance
      org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces

      These cause search times to increase from a few milliseconds to up to 2 seconds when doing 500 concurrent searches on the same in memory index. Note: That the index is not being updates at all, so not refresh methods are called at any stage.

      Some questions:
      Why do we need synchronization here?
      There must be a non-lockable solution for these, they basically cause lucene to be ok for single thread applications but disastrous for any concurrent implementation.

      I'll do some experiments by removing the synchronization from the methods of these classes.

      Attachments

        1. profile_1_a.png
          133 kB
          Gerrit Jansen van Vuuren
        2. profile_1_b.png
          143 kB
          Gerrit Jansen van Vuuren
        3. profile_1_c.png
          162 kB
          Gerrit Jansen van Vuuren
        4. profile_1_d.png
          171 kB
          Gerrit Jansen van Vuuren
        5. profile_2_a.png
          128 kB
          Gerrit Jansen van Vuuren
        6. profile_2_b.png
          160 kB
          Gerrit Jansen van Vuuren
        7. profile_2_c.png
          135 kB
          Gerrit Jansen van Vuuren
        8. App.java
          3 kB
          Gerrit Jansen van Vuuren
        9. LUCENE-3653-VirtualMethod+AttributeSource.patch
          17 kB
          Uwe Schindler
        10. LUCENE-3653-VirtualMethod+AttributeSource.patch
          17 kB
          Uwe Schindler
        11. lucene-unsync.diff
          17 kB
          Gerrit Jansen van Vuuren
        12. LUCENE-3653-VirtualMethod+AttributeSource.patch
          21 kB
          Uwe Schindler
        13. LUCENE-3653.patch
          2 kB
          Simon Willnauer
        14. LUCENE-3653.patch-BiasedLockingStartupDelay_3.png
          129 kB
          Gerrit Jansen van Vuuren
        15. LUCENE-3653.patch-BiasedLockingStartupDelay_2.png
          129 kB
          Gerrit Jansen van Vuuren
        16. LUCENE-3653.patch-BiasedLockingStartupDelay_1.png
          130 kB
          Gerrit Jansen van Vuuren
        17. Threads-LUCENE-3653.patch.png
          128 kB
          Gerrit Jansen van Vuuren
        18. LUCENE-3653-no-sync.png
          33 kB
          Simon Willnauer
        19. LUCENE-3653-sync-.png
          32 kB
          Simon Willnauer
        20. LUCENE-3653.patch
          0.9 kB
          Simon Willnauer

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            simonw Simon Willnauer
            gerritjvv Gerrit Jansen van Vuuren
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment