Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9764

Design a memory efficient DocSet if a query returns all docs

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      In some use cases, particularly use cases with time series data, using collection alias and partitioning data into multiple small collections using timestamp, a filter query can match all documents in a collection. Currently BitDocSet is used which contains a large array of long integers with every bits set to 1. After querying, the resulted DocSet saved in filter cache is large and becomes one of the main memory consumers in these use cases.

      For example. suppose a Solr setup has 14 collections for data in last 14 days, each collection with one day of data. A filter query for last one week data would result in at least six DocSet in filter cache which matches all documents in six collections respectively.

      This is to design a new DocSet that is memory efficient for such a use case. The new DocSet removes the large array, reduces memory usage and GC pressure without losing advantage of large filter cache.

      In particular, for use cases when using time series data, collection alias and partition data into multiple small collections using timestamp, the gain can be large.

      For further optimization, it may be helpful to design a DocSet with run length encoding. Thanks Mostafa Mokhtar for suggestion.

        Attachments

        1. SOLR_9764_no_cloneMe.patch
          13 kB
          David Smiley
        2. SOLR-9764.patch
          19 kB
          Yonik Seeley
        3. SOLR-9764.patch
          17 kB
          Yonik Seeley
        4. SOLR-9764.patch
          15 kB
          Michael Sun
        5. SOLR-9764.patch
          12 kB
          Michael Sun
        6. SOLR-9764.patch
          12 kB
          Michael Sun
        7. SOLR-9764.patch
          9 kB
          Michael Sun
        8. SOLR-9764.patch
          9 kB
          Michael Sun
        9. SOLR-9764.patch
          3 kB
          Michael Sun

          Issue Links

            Activity

              People

              • Assignee:
                yseeley@gmail.com Yonik Seeley
                Reporter:
                michael.sun Michael Sun
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: