Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9599

DocValues performance regression with new iterator API

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: master (7.0)
    • Fix Version/s: master (7.0)
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      I did a quick performance comparison of faceting indexed fields (i.e. docvalues are not stored) using method=dv before and after the new docvalues iterator went in (LUCENE-7407).

      5M document index, 21 segments, single valued string fields w/ no missing values.

      field cardinality new_time / old_time
      10 2.01
      1000 2.02
      10000 1.85
      100000 1.56
      1000000 1.31

      So unfortunately, often twice as slow.

      See followup messages for tests using real docvalues as well.

        Issue Links

          Activity

          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Here's the form of the request used (tests were run w/ logging at WARN level of course):

          2016-10-05 00:25:24.042 INFO  (qtp110456297-16) [   x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&json.facet={x:{method:dv,+type:terms,field:s10_s,limit:5}}&rows=0&wt=javabin} hits=4993847 status=0 QTime=174
          
          Show
          yseeley@gmail.com Yonik Seeley added a comment - Here's the form of the request used (tests were run w/ logging at WARN level of course): 2016-10-05 00:25:24.042 INFO (qtp110456297-16) [ x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/select params={q=*:*&json.facet={x:{method:dv,+type:terms,field:s10_s,limit:5}}&rows=0&wt=javabin} hits=4993847 status=0 QTime=174
          Hide
          yseeley@gmail.com Yonik Seeley added a comment - - edited

          A quick test of the same fields in the same index shows hits to sorting and function queries as well.

          With a quick manual test, this was 51% faster previously (before LUCENE-7407) for fieldcache fields, and 37% faster for docvalue fields:

          http://localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW&fl=id&sort=s10_s%20desc,%20s100_s%20desc,%20s1000_s%20desc
          

          And this was 78% faster for fieldcache, and 29% faster for docvalues:

          http://localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW%20{!func%20v=$vv}&fl=id&vv=add(exists(s10_s),exists(s100_s),exists(s1000_s))
          

          Integer field function queries were 75% faster, and docvalues were 50% faster:

          http://localhost:8983/solr/collection1/query?q=mydate_dt:NOW%20{!func%20v=$vv}&fl=id&vv=add(s10_i,s100_i,s1000_i,s10000_i)
          
          Show
          yseeley@gmail.com Yonik Seeley added a comment - - edited A quick test of the same fields in the same index shows hits to sorting and function queries as well. With a quick manual test, this was 51% faster previously (before LUCENE-7407 ) for fieldcache fields, and 37% faster for docvalue fields: http: //localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW&fl=id&sort=s10_s%20desc,%20s100_s%20desc,%20s1000_s%20desc And this was 78% faster for fieldcache, and 29% faster for docvalues: http: //localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW%20{!func%20v=$vv}&fl=id&vv=add(exists(s10_s),exists(s100_s),exists(s1000_s)) Integer field function queries were 75% faster, and docvalues were 50% faster: http: //localhost:8983/solr/collection1/query?q=mydate_dt:NOW%20{!func%20v=$vv}&fl=id&vv=add(s10_i,s100_i,s1000_i,s10000_i)
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          I tried some docValue fields this time instead of the fieldcache:

          field cardinality new_time / old_time
          10 1.29
          1000 1.23
          10000 1.24
          100000 1.34
          1000000 1.09
          Show
          yseeley@gmail.com Yonik Seeley added a comment - I tried some docValue fields this time instead of the fieldcache: field cardinality new_time / old_time 10 1.29 1000 1.23 10000 1.24 100000 1.34 1000000 1.09
          Hide
          yseeley@gmail.com Yonik Seeley added a comment - - edited

          Another docvalues faceting test, this time including the current lucene/solr code + lucene70 codec (as of 10/17)
          This test used 10M documents and single valued string fields with 20% of the values missing (i.e. 80% of docs have a value for any given field). 4 concurrent request threads were used with a 4 core CPU.
          Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

          This is a table of new_time/old_time, with old_time being an old docvalues index with old code (as of 9/09) before the docvalues iterator cutover:

          field cardinality 9/09 code with 9/09 index 10/17 code with 9/09 index 10/17 code with 10/17 index
          10 1.00 1.39 1.41
          100 1.00 1.38 1.46
          1000 1.00 1.39 1.42
          10000 1.00 1.35 1.45

          So it looks like we're currently over 40% slower in general for faceting on single valued docvalue fields that have some values missing.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - - edited Another docvalues faceting test, this time including the current lucene/solr code + lucene70 codec (as of 10/17) This test used 10M documents and single valued string fields with 20% of the values missing (i.e. 80% of docs have a value for any given field). 4 concurrent request threads were used with a 4 core CPU. Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments. This is a table of new_time/old_time, with old_time being an old docvalues index with old code (as of 9/09) before the docvalues iterator cutover: field cardinality 9/09 code with 9/09 index 10/17 code with 9/09 index 10/17 code with 10/17 index 10 1.00 1.39 1.41 100 1.00 1.38 1.46 1000 1.00 1.39 1.42 10000 1.00 1.35 1.45 So it looks like we're currently over 40% slower in general for faceting on single valued docvalue fields that have some values missing.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment - - edited

          Here's a more thorough/accurate (not by-hand) sorting test on the same index above (docvalue fields, 10M docs, 80% value density).
          Base query = *:*, 4 query threads, finding top 10 documents.
          Representative query:

          q={!cache%3Dfalse}*:*&fl=id&sort=s10_i+asc,s1000000_i+desc&rows=10&wt=javabin
          

          Results below in new_time/old_time:

          one string sort field (fields have cardinality of 10,100,1000,10000,100000, or 1000000): 1.20 
          2 string sort fields, first has cardinality of 10 or 100: 1.27
          2 integer sort fields, first has cardinality of 10 or 100: 1.46
          
          Show
          yseeley@gmail.com Yonik Seeley added a comment - - edited Here's a more thorough/accurate (not by-hand) sorting test on the same index above (docvalue fields, 10M docs, 80% value density). Base query = *:*, 4 query threads, finding top 10 documents. Representative query: q={!cache%3Dfalse}*:*&fl=id&sort=s10_i+asc,s1000000_i+desc&rows=10&wt=javabin Results below in new_time/old_time: one string sort field (fields have cardinality of 10,100,1000,10000,100000, or 1000000): 1.20 2 string sort fields, first has cardinality of 10 or 100: 1.27 2 integer sort fields, first has cardinality of 10 or 100: 1.46
          Hide
          otis Otis Gospodnetic added a comment -

          Yonik Seeley all sub-tasks seem to be done/resolved.... should this then be resolved, too?

          Show
          otis Otis Gospodnetic added a comment - Yonik Seeley all sub-tasks seem to be done/resolved.... should this then be resolved, too?
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          No, the list of subtasks isn't comprehensive or anything... I added some as needed so I could commit some progress.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - No, the list of subtasks isn't comprehensive or anything... I added some as needed so I could commit some progress.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          I've started performance testing the new iterator API as part of the Apache Calcite integration. In particular I've been testing the ExportWriter's sort and export performance with String fields. So far the performance numbers have been comparable to the random access API.

          The types of Streaming Expressions I've been running look like this:

          null(search(enron, q="*:*", fl="to", sort="to desc", qt="/export"))
          

          This will export and sort all the values in the "to" field in the enron email data set. The null function simply drops are the tuples so we fully isolate the performance of the /export.

          I've been using the Direct doc values format for the test, but I'll reindex with the default docValues format this week. But typically my advice to anyone that wants to maximize streaming performance is to use Direct docValues.

          Currently both the old and new docValues API's perform this operation in 400 ms.

          I'll continue to update this thread as I test numerics and different docValues formats, and also increase the size of indexes.

          I'll also be testing the performance of field collapsing.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited I've started performance testing the new iterator API as part of the Apache Calcite integration. In particular I've been testing the ExportWriter's sort and export performance with String fields. So far the performance numbers have been comparable to the random access API. The types of Streaming Expressions I've been running look like this: null (search(enron, q= "*:*" , fl= "to" , sort= "to desc" , qt= "/export" )) This will export and sort all the values in the "to" field in the enron email data set. The null function simply drops are the tuples so we fully isolate the performance of the /export. I've been using the Direct doc values format for the test, but I'll reindex with the default docValues format this week. But typically my advice to anyone that wants to maximize streaming performance is to use Direct docValues. Currently both the old and new docValues API's perform this operation in 400 ms. I'll continue to update this thread as I test numerics and different docValues formats, and also increase the size of indexes. I'll also be testing the performance of field collapsing.

            People

            • Assignee:
              Unassigned
              Reporter:
              yseeley@gmail.com Yonik Seeley
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development