Solr
  1. Solr
  2. SOLR-7799

LukeRequestHandler too slow when there are many fields

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.2
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      /admin/luke responds very slowly when there are many fields in the index. The issue is with getFirstLiveDoc() spinning its wheels a lot to get the index flags.

      1. SOLR-7799.patch
        3 kB
        Erik Hatcher

        Activity

        Hide
        Erik Hatcher added a comment -

        Here's a patch that adds an "includeIndexFieldFlags" option, defaulting to true for backwards compatibility. Setting this flag to false greatly speeds up the request. The schema flags are still included as those aren't a performance issue.

        Show
        Erik Hatcher added a comment - Here's a patch that adds an "includeIndexFieldFlags" option, defaulting to true for backwards compatibility. Setting this flag to false greatly speeds up the request. The schema flags are still included as those aren't a performance issue.
        Hide
        Erik Hatcher added a comment -

        getFirstLiveDoc() could probably be sped up, but those index field flags aren't really that useful or needed in our case so I took the easy way out with the initial patch to just avoid going there.

        Show
        Erik Hatcher added a comment - getFirstLiveDoc() could probably be sped up, but those index field flags aren't really that useful or needed in our case so I took the easy way out with the initial patch to just avoid going there.
        Hide
        Erik Hatcher added a comment -

        I tested with two indexes, 4k docs with (the same) 4k fields each. One index has the fields as long, the other as text_general.

        lots of long fields:
        /solr/lotsafields_long/admin/luke?numTerms=0&includeIndexFieldFlags=true&indent=on: 801ms
        /solr/lotsafields_long/admin/luke?numTerms=0&includeIndexFieldFlags=false&indent=on: 13ms
        
        lots of text_general fields:
        /solr/lotsafields_txt/admin/luke?numTerms=0&includeIndexFieldFlags=true&indent=on: 2169ms
        /solr/lotsafields_txt/admin/luke?numTerms=0&includeIndexFieldFlags=false&indent=on: 16ms
        
        Show
        Erik Hatcher added a comment - I tested with two indexes, 4k docs with (the same) 4k fields each. One index has the fields as long, the other as text_general. lots of long fields: /solr/lotsafields_long/admin/luke?numTerms=0&includeIndexFieldFlags= true &indent=on: 801ms /solr/lotsafields_long/admin/luke?numTerms=0&includeIndexFieldFlags= false &indent=on: 13ms lots of text_general fields: /solr/lotsafields_txt/admin/luke?numTerms=0&includeIndexFieldFlags= true &indent=on: 2169ms /solr/lotsafields_txt/admin/luke?numTerms=0&includeIndexFieldFlags= false &indent=on: 16ms
        Hide
        ASF subversion and git services added a comment -

        Commit 1693935 from Erik Hatcher in branch 'dev/trunk'
        [ https://svn.apache.org/r1693935 ]

        SOLR-7799: Added includeIndexFieldFlags to /admin/luke

        Show
        ASF subversion and git services added a comment - Commit 1693935 from Erik Hatcher in branch 'dev/trunk' [ https://svn.apache.org/r1693935 ] SOLR-7799 : Added includeIndexFieldFlags to /admin/luke
        Hide
        ASF subversion and git services added a comment -

        Commit 1693943 from Erik Hatcher in branch 'dev/trunk'
        [ https://svn.apache.org/r1693943 ]

        SOLR-7799: fix mistake of docs count for terms being inside the new if condition inadvertently

        Show
        ASF subversion and git services added a comment - Commit 1693943 from Erik Hatcher in branch 'dev/trunk' [ https://svn.apache.org/r1693943 ] SOLR-7799 : fix mistake of docs count for terms being inside the new if condition inadvertently
        Hide
        ASF subversion and git services added a comment -

        Commit 1693946 from Erik Hatcher in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1693946 ]

        SOLR-7799: Added includeIndexFieldFlags to /admin/luke

        Show
        ASF subversion and git services added a comment - Commit 1693946 from Erik Hatcher in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1693946 ] SOLR-7799 : Added includeIndexFieldFlags to /admin/luke
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk close for 5.3.0 release

        Show
        Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

          People

          • Assignee:
            Erik Hatcher
            Reporter:
            Erik Hatcher
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development