Solr
  1. Solr
  2. SOLR-6259

Performance issue with large number of fields and values when using copyFields

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.8.1
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      When you have schema with a large enough number of fields (in my case around 250 fields) and you use copyFields to populate a number of fields (very few in my case 3-4) you see a severe degradation in the performance of ingestion.

      Tracking this down using a profiler found that in the lucene Document.getField() was using 87% of all CPU time. As it turns out getField() does an iteration over the list of fields in the Document returning the field if the name matches.. Anyway in the case of copyFields with lots of values getField() gets called alot...

      1. SOLR-6259.patch
        5 kB
        Steven Bower

        Issue Links

          Activity

          Hide
          Steven Bower added a comment -

          Attached patch that fixes this issue. Basically combination of tracking which fields are used in a hashset and by moving repeated checking from inner loops to further out loops.

          Show
          Steven Bower added a comment - Attached patch that fixes this issue. Basically combination of tracking which fields are used in a hashset and by moving repeated checking from inner loops to further out loops.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Steven.

          Committed r1611852 on trunk and r1611853 on branch_4x.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Steven. Committed r1611852 on trunk and r1611853 on branch_4x.
          Hide
          Shingo Sasaki added a comment -

          I think that the bug in multivalue field happened by this fix.

          Please read that issue SOLR-7335.

          Show
          Shingo Sasaki added a comment - I think that the bug in multivalue field happened by this fix. Please read that issue SOLR-7335 .

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Steven Bower
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development