Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14428

FuzzyQuery has severe memory usage in 8.5

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 8.5, 8.5.1
    • 8.6
    • None
    • None

    Description

      I sent this to the mailing list

      I'm moving from 8.3.1 to 8.5.1, and started getting Out Of Memory Errors while running our normal tests. After profiling it was clear that the majority of the heap was allocated through FuzzyQuery.
      LUCENE-9068 moved construction of the automata from the FuzzyTermsEnum to the FuzzyQuery's constructor.

      I created a little test ( FuzzyHammer.java ) that fires off fuzzy queries from random UUID strings for 5 minutes

      FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"
      

      When running against a vanilla Solr 8.31 and 8.4.1 there is no problem, while the memory usage has increased drastically on 8.5.0 and 8.5.1.

      Comparison of heap usage while running the attached test against Solr 8.3.1 and 8.5.1 with a single (empty) shard and 4GB heap:

      And with 4 shards on 8.4.1 and 8.5.0:

      I'm guessing that the memory might be being leaked if the FuzzyQuery objects are referenced from the cache, while the FuzzyTermsEnum would not have been.

      Query Result Cache on 8.5.1:

      ~316mb in the cache

      QRC on 8.3.1

      <1mb

      With an empty cache, running this query field_s:e41848af85d24ac197c71db6888e17bc~2 results in the following memory allocation

      8.3.1: CACHE.searcher.queryResultCache.ramBytesUsed:      1520
      8.5.1: CACHE.searcher.queryResultCache.ramBytesUsed:    648855
      

      ~1 gives 98253 and ~0 gives 6339 on 8.5.1. 8.3.1 is constant at 1520

      Attachments

        1. FuzzyHammer.java
          1 kB
          cjcowie
        2. image-2020-04-23-09-18-06-070.png
          35 kB
          cjcowie
        3. image-2020-04-24-20-09-31-179.png
          47 kB
          cjcowie
        4. screenshot-2.png
          31 kB
          cjcowie
        5. screenshot-3.png
          29 kB
          cjcowie
        6. screenshot-4.png
          29 kB
          cjcowie
        7. SOLR-14428-WeakReferences.patch
          5 kB
          cjcowie

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            romseygeek Alan Woodward Assign to me
            colvinco Colvin Cowie
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 10m
              10m

              Slack

                Issue deployment