Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7407

Explore switching doc values to an iterator API

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: master (7.0)
    • Component/s: None
    • Labels:
    • Lucene Fields:
      New

      Description

      I think it could be compelling if we restricted doc values to use an
      iterator API at read time, instead of the more general random access
      API we have today:

      • It would make doc values disk usage more of a "you pay for what
        what you actually use", like postings, which is a compelling
        reduction for sparse usage.
      • I think codecs could compress better and maybe speed up decoding
        of doc values, even in the non-sparse case, since the read-time
        API is more restrictive "forward only" instead of random access.
      • We could remove getDocsWithField entirely, since that's
        implicit in the iteration, and the awkward "return 0 if the
        document didn't have this field" would go away.
      • We can remove the annoying thread locals we must make today in
        CodecReader, and close the trappy "I accidentally shared a
        single XXXDocValues instance across threads", since an iterator is
        inherently "use once".
      • We could maybe leverage the numerous optimizations we've done for
        postings over time, since the two problems ("iterate over doc ids
        and store something interesting for each") are very similar.

      This idea has come up many in the past, e.g. LUCENE-7253 is a recent
      example, and very early iterations of doc values started with exactly
      this

      However, it's a truly enormous change, likely 7.0 only. Or maybe we
      could have the new iterator APIs also ported to 6.x side by side with
      the deprecate existing random-access APIs.

      1. LUCENE-7407.patch
        1.20 MB
        Michael McCandless

        Issue Links

          Activity

          Hide
          mikemccand Michael McCandless added a comment -

          I've been prototyping this, and I just pushed my current state to:

          https://github.com/mikemccand/lucene-solr/tree/dv_iterators

          I just added a NumericDocValuesIterator which extends DISI,
          adding a public long longValue() method. So I can make baby
          steps, I added getNumericIterator side by side with getNumeric
          in DocValuesProducer with a stupid default implementation. This
          lets me gradually cutover doc-values consumers to see what interesting
          cases emerge. Index time sorting was one interesting case, and block
          join sorting is another. I'm sure there are others.

          I also pushed up throws IOException; I think it's silly we mask
          this today in doc values.

          I'll keep cutting numerics over to assess feasibility...

          Show
          mikemccand Michael McCandless added a comment - I've been prototyping this, and I just pushed my current state to: https://github.com/mikemccand/lucene-solr/tree/dv_iterators I just added a NumericDocValuesIterator which extends DISI, adding a public long longValue() method. So I can make baby steps, I added getNumericIterator side by side with getNumeric in DocValuesProducer with a stupid default implementation. This lets me gradually cutover doc-values consumers to see what interesting cases emerge. Index time sorting was one interesting case, and block join sorting is another. I'm sure there are others. I also pushed up throws IOException ; I think it's silly we mask this today in doc values. I'll keep cutting numerics over to assess feasibility...
          Hide
          jpountz Adrien Grand added a comment -

          Wow, thanks for getting this started! I had a quick look at the diff, it looks great. One minor suggestion I have would be to make the API return a DocIdSetIterator (like Scorer) rather than extend it. In LUCENE-6919 this helped avoid having to wrap all the time, which also made performance better. In the case of doc values, codecs could have one way to get an iterator that would be reused by all doc value types (NUMERIC, SORTED, ...) for instance.

          Show
          jpountz Adrien Grand added a comment - Wow, thanks for getting this started! I had a quick look at the diff, it looks great. One minor suggestion I have would be to make the API return a DocIdSetIterator (like Scorer) rather than extend it. In LUCENE-6919 this helped avoid having to wrap all the time, which also made performance better. In the case of doc values, codecs could have one way to get an iterator that would be reused by all doc value types (NUMERIC, SORTED, ...) for instance.
          Hide
          otis Otis Gospodnetic added a comment -

          Can I label this with #AWESOME!!! ? Could Adrien's LUCENE-6928 piggyback on this?

          Show
          otis Otis Gospodnetic added a comment - Can I label this with #AWESOME!!! ? Could Adrien's LUCENE-6928 piggyback on this?
          Hide
          mikemccand Michael McCandless added a comment -

          One minor suggestion I have would be to make the API return a DocIdSetIterator (like Scorer) rather than extend it.

          Hmm, this seems somewhat awkward?

          It would mean to iterate doc values, you would need to hold onto two
          classes: the iterator you pulled, and the parent class you pulled it
          from. I think it's also odd that the iterator is altering state in another
          class.

          I realize we did this for performance reasons for Scorer, but it's
          not great to compromise the API so much for performance unless the
          performance change is really drastic. Could that be the case here?

          Show
          mikemccand Michael McCandless added a comment - One minor suggestion I have would be to make the API return a DocIdSetIterator (like Scorer) rather than extend it. Hmm, this seems somewhat awkward? It would mean to iterate doc values, you would need to hold onto two classes: the iterator you pulled, and the parent class you pulled it from. I think it's also odd that the iterator is altering state in another class. I realize we did this for performance reasons for Scorer , but it's not great to compromise the API so much for performance unless the performance change is really drastic. Could that be the case here?
          Hide
          mikemccand Michael McCandless added a comment -

          Just a quick progress update here: I've managed to cut over 4 (Numeric, Binary, Sorted, norms) of the 6 DV classes ... working on SortedSet now, and then finally SortedNumeric.

          Then I need to remove the crazy bridge classes, then remove the old (non-iterator) classes, then rename the iterator classes back to the original names.

          Net/net I think can work, but I still have plenty of nocommits to sort out.

          Show
          mikemccand Michael McCandless added a comment - Just a quick progress update here: I've managed to cut over 4 (Numeric, Binary, Sorted, norms) of the 6 DV classes ... working on SortedSet now, and then finally SortedNumeric. Then I need to remove the crazy bridge classes, then remove the old (non-iterator) classes, then rename the iterator classes back to the original names. Net/net I think can work, but I still have plenty of nocommits to sort out.
          Hide
          dsmiley David Smiley added a comment -

          Hi; is this API change only at the Codec level or is it higher level as well?

          Show
          dsmiley David Smiley added a comment - Hi; is this API change only at the Codec level or is it higher level as well?
          Hide
          mikemccand Michael McCandless added a comment -

          Higher level as well: rather than a random access NumericDocValues, you have a NumericDocValuesIterator extending DocIdSetIterator and just adding a longValue method, for example.

          Show
          mikemccand Michael McCandless added a comment - Higher level as well: rather than a random access NumericDocValues , you have a NumericDocValuesIterator extending DocIdSetIterator and just adding a longValue method, for example.
          Hide
          otis Otis Gospodnetic added a comment -

          Once these changes are made do you think one will be able to just replace the Lucene jar in e.g. ES 5.x?

          Show
          otis Otis Gospodnetic added a comment - Once these changes are made do you think one will be able to just replace the Lucene jar in e.g. ES 5.x?
          Hide
          mikemccand Michael McCandless added a comment -

          Sorry, I don't think so Otis Gospodnetic: this is a major change, I think it will only be for Lucene 7.0?

          Show
          mikemccand Michael McCandless added a comment - Sorry, I don't think so Otis Gospodnetic : this is a major change, I think it will only be for Lucene 7.0?
          Hide
          mikemccand Michael McCandless added a comment -

          I think this nearly ready! I've fixed all nocommits, but ant precommit is a bit angry still... I'll fix before pushing.

          I'm attaching an applyable patch vs. current master.

          All doc values usage has been switched to iterators instead of random access, and getDocsWithField is gone.

          I've done very little to improve the default codec to take advantage of this. I think there is a lot of fun improvements we can make here, in follow-on issues, so that e.g. LUCENE-7253 (merging of sparse doc values fields) is fixed.

          To write doc values we now pass a DocValuesProducer (instead of N Iterables), and I created legacy deprecated bridge classes (LegacyDocValuesIterables) to turns these back into Iterables for existing codecs.

          I also created legacy bridge classes to turn random access DVs into the new iterators.

          Show
          mikemccand Michael McCandless added a comment - I think this nearly ready! I've fixed all nocommits, but ant precommit is a bit angry still... I'll fix before pushing. I'm attaching an applyable patch vs. current master. All doc values usage has been switched to iterators instead of random access, and getDocsWithField is gone. I've done very little to improve the default codec to take advantage of this. I think there is a lot of fun improvements we can make here, in follow-on issues, so that e.g. LUCENE-7253 (merging of sparse doc values fields) is fixed. To write doc values we now pass a DocValuesProducer (instead of N Iterables), and I created legacy deprecated bridge classes ( LegacyDocValuesIterables ) to turns these back into Iterables for existing codecs. I also created legacy bridge classes to turn random access DVs into the new iterators.
          Hide
          dsmiley David Smiley added a comment -

          Awesome progress Mike! It's nice to see we don't have to worry anymore about the '0' value being possibly non-existent.

          It appears advance(docId) is inherited from DISI and thus it must be called with sequential docIDs. Did you run into any spot in the codebase that didn't advance sequentially and so you had to do something different?

          Show
          dsmiley David Smiley added a comment - Awesome progress Mike! It's nice to see we don't have to worry anymore about the '0' value being possibly non-existent. It appears advance(docId) is inherited from DISI and thus it must be called with sequential docIDs. Did you run into any spot in the codebase that didn't advance sequentially and so you had to do something different?
          Hide
          mikemccand Michael McCandless added a comment -

          It's nice to see we don't have to worry anymore about the '0' value being possibly non-existent.

          Right!

          It appears advance(docId) is inherited from DISI and thus it must be called with sequential docIDs. Did you run into any spot in the codebase that didn't advance sequentially and so you had to do something different?

          I was surprised how few places didn't go in order ...

          E.g., index sorting was tricky, since it very much relied on random access.

          A number of tests also accessed docIDs that came back in hits which of course may not be "in order".

          Show
          mikemccand Michael McCandless added a comment - It's nice to see we don't have to worry anymore about the '0' value being possibly non-existent. Right! It appears advance(docId) is inherited from DISI and thus it must be called with sequential docIDs. Did you run into any spot in the codebase that didn't advance sequentially and so you had to do something different? I was surprised how few places didn't go in order ... E.g., index sorting was tricky, since it very much relied on random access. A number of tests also accessed docIDs that came back in hits which of course may not be "in order".
          Hide
          otis Otis Gospodnetic added a comment -

          there is a lot of fun improvements we can make here, in follow-on issues, so that e.g. LUCENE-7253 (merging of sparse doc values fields) is fixed.

          So LUCENE-7253 is where the new Codec work for trunk will go?
          Did you maybe create the other issues you mentioned? Asking because I'm curious what you have in mind and so I can link+watch.

          Show
          otis Otis Gospodnetic added a comment - there is a lot of fun improvements we can make here, in follow-on issues, so that e.g. LUCENE-7253 (merging of sparse doc values fields) is fixed. So LUCENE-7253 is where the new Codec work for trunk will go? Did you maybe create the other issues you mentioned? Asking because I'm curious what you have in mind and so I can link+watch.
          Hide
          mikemccand Michael McCandless added a comment -

          Did you maybe create the other issues you mentioned?

          I haven't created a follow-on issue yet ... I will soon. I think it should be separate from LUCENE-7253.

          Show
          mikemccand Michael McCandless added a comment - Did you maybe create the other issues you mentioned? I haven't created a follow-on issue yet ... I will soon. I think it should be separate from LUCENE-7253 .
          Hide
          dsmiley David Smiley added a comment -

          BTW one way that this commit could have been less massive it to split out the "throws IOException" additions as one change, and then subsequently get to the meat of the work here. Any way, that's water under the bridge.

          Show
          dsmiley David Smiley added a comment - BTW one way that this commit could have been less massive it to split out the "throws IOException" additions as one change, and then subsequently get to the meat of the work here. Any way, that's water under the bridge.
          Hide
          mikemccand Michael McCandless added a comment -

          BTW one way that this commit could have been less massive it to split out the "throws IOException" additions as one change

          That's a good point

          Show
          mikemccand Michael McCandless added a comment - BTW one way that this commit could have been less massive it to split out the "throws IOException" additions as one change That's a good point
          Hide
          mikemccand Michael McCandless added a comment -

          I opened LUCENE-7457 for the (important!) follow-on.

          Show
          mikemccand Michael McCandless added a comment - I opened LUCENE-7457 for the (important!) follow-on.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7377d0ef9ea8fa9e2aa9a3ccb1249703d8d1d813 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7377d0e ]

          LUCENE-7407: fix stale javadocs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7377d0ef9ea8fa9e2aa9a3ccb1249703d8d1d813 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7377d0e ] LUCENE-7407 : fix stale javadocs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 53dd74bd870437ecee0483096e3ef5669d844e57 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dd74b ]

          LUCENE-7407: fix stale javadocs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 53dd74bd870437ecee0483096e3ef5669d844e57 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53dd74b ] LUCENE-7407 : fix stale javadocs
          Hide
          steve_rowe Steve Rowe added a comment -

          This change is implicated in a Solr test failure: SOLR-9582

          Show
          steve_rowe Steve Rowe added a comment - This change is implicated in a Solr test failure: SOLR-9582
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 001a3ca55b30656e0e42f612d927a7923f5370e9 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=001a3ca ]

          LUCENE-7407: speed up iterating norms a bit by having default codec implement the iterator directly

          Show
          jira-bot ASF subversion and git services added a comment - Commit 001a3ca55b30656e0e42f612d927a7923f5370e9 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=001a3ca ] LUCENE-7407 : speed up iterating norms a bit by having default codec implement the iterator directly
          Hide
          jpountz Adrien Grand added a comment -

          It looks like this last change helped significantly: http://people.apache.org/~mikemccand/lucenebench/Term.html

          Show
          jpountz Adrien Grand added a comment - It looks like this last change helped significantly: http://people.apache.org/~mikemccand/lucenebench/Term.html
          Hide
          mikemccand Michael McCandless added a comment -

          It looks like this last change helped significantl

          Nice

          Show
          mikemccand Michael McCandless added a comment - It looks like this last change helped significantl Nice
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          It seems like this has had a rather large impact on what was supposed to be the common case: dense doc values. See SOLR-9599 for some performance numbers, currently standing at 40% slower for faceting as of today, but also impacting all other docvalue uses also tested (sorting and function queries).
          Perhaps we should have both a random access API as well as an iterator API?

          What do people thing about the right path forward?

          Show
          yseeley@gmail.com Yonik Seeley added a comment - It seems like this has had a rather large impact on what was supposed to be the common case: dense doc values. See SOLR-9599 for some performance numbers, currently standing at 40% slower for faceting as of today, but also impacting all other docvalue uses also tested (sorting and function queries). Perhaps we should have both a random access API as well as an iterator API? What do people thing about the right path forward?
          Hide
          jpountz Adrien Grand added a comment -

          I agree there are things to be improved there (see LUCENE-7462 too).

          The comparison might not be entirely fair to the new API since the Lucene54 format was really designed with the old random-access API in mind. I'm wondering how much we can get back by more naturally implementing the new API. But I suspect we will have to do more to get back to performance that is close to what we had before, at least in the dense case. To me there are two ways that we can do it:

          • adjust the iterator API of doc values to require less search-time work. Because of the advance() semantics, we currently need to guard all value accesses under something that looks like this:
                  int curDocID = docTerms.docID();
                  if (doc > curDocID) {
                    curDocID = docTerms.advance(doc);
                  }
                  if (doc == curDocID) {
                    // handle value
                  } else {
                    // handle missing value
                  }
            

            (copied from FieldComparator). The advance() semantics both return the next document that has a value, which we never need at search time so this is an unnecessary effort from the codec, and also require that the target is strictly beyond the current document, which prevents from calling advance(doc) blindly: we need to check whether the iterator is on the current document or beyond already. Maybe we could have instead something like an advanceExact(target) method that would only advance to the target document and return whether it has a value.

          • have a 2nd DV API that looks like the old API (with the additional constraint that doc ids need to be consumed in order) and helpers in the DocValues class to convert from dv producers with an iterator API to this random-access API (the LUCENE-7462 proposal). When the codec specializes the dense case (which would be always the case with the default codec), the conversion would only unwrap the iterator to return a random-access API. And otherwise it would wrap in order to check the current doc ID and advance if necessary (like almost every consumer of the doc values APIs needs to do now in master).
          Show
          jpountz Adrien Grand added a comment - I agree there are things to be improved there (see LUCENE-7462 too). The comparison might not be entirely fair to the new API since the Lucene54 format was really designed with the old random-access API in mind. I'm wondering how much we can get back by more naturally implementing the new API. But I suspect we will have to do more to get back to performance that is close to what we had before, at least in the dense case. To me there are two ways that we can do it: adjust the iterator API of doc values to require less search-time work. Because of the advance() semantics, we currently need to guard all value accesses under something that looks like this: int curDocID = docTerms.docID(); if (doc > curDocID) { curDocID = docTerms.advance(doc); } if (doc == curDocID) { // handle value } else { // handle missing value } (copied from FieldComparator ). The advance() semantics both return the next document that has a value, which we never need at search time so this is an unnecessary effort from the codec, and also require that the target is strictly beyond the current document, which prevents from calling advance(doc) blindly: we need to check whether the iterator is on the current document or beyond already. Maybe we could have instead something like an advanceExact(target) method that would only advance to the target document and return whether it has a value. have a 2nd DV API that looks like the old API (with the additional constraint that doc ids need to be consumed in order) and helpers in the DocValues class to convert from dv producers with an iterator API to this random-access API (the LUCENE-7462 proposal). When the codec specializes the dense case (which would be always the case with the default codec), the conversion would only unwrap the iterator to return a random-access API. And otherwise it would wrap in order to check the current doc ID and advance if necessary (like almost every consumer of the doc values APIs needs to do now in master).
          Hide
          mikemccand Michael McCandless added a comment -

          Hmm, in Lucene's nightly perf tests, the TermDateFacets got only a bit slower (~11%), not 40% slower. Yonik, can you give more details on your benchmark so others can run it?

          We should also look at the Solr faceting code to see if it can be improved on how it's using the DV iterators; I just did a quick cutover to the iterator API for this issue, and maybe there's something inefficient there.

          I don't think Lucene should have two APIs (LUCENE-7462). This will lead to too much bifurcation of the code that consumes doc values. That's the wrong tradeoff, and we shouldn't let performance mess up our APIs that heavily.

          That said, I think an advanceExact would be a good middle ground, if we show it can in fact help performance. We could make a simple default impl in either DISI or maybe a new base class for all doc values impls.

          Show
          mikemccand Michael McCandless added a comment - Hmm, in Lucene's nightly perf tests, the TermDateFacets got only a bit slower (~11%), not 40% slower. Yonik, can you give more details on your benchmark so others can run it? We should also look at the Solr faceting code to see if it can be improved on how it's using the DV iterators; I just did a quick cutover to the iterator API for this issue, and maybe there's something inefficient there. I don't think Lucene should have two APIs ( LUCENE-7462 ). This will lead to too much bifurcation of the code that consumes doc values. That's the wrong tradeoff, and we shouldn't let performance mess up our APIs that heavily. That said, I think an advanceExact would be a good middle ground, if we show it can in fact help performance. We could make a simple default impl in either DISI or maybe a new base class for all doc values impls.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Hmm, in Lucene's nightly perf tests, the TermDateFacets got only a bit slower (~11%), not 40% slower. Yonik, can you give more details on your benchmark so others can run it?

          Amdahl's law? My tests are probably just isolating the docValues performance more. These are full-stack tests (on both sides?)... so it may be that TermDateFacets spends less of it's execution time actually retrieving docValues, and has more bottlenecks elsewhere. I'm also effectively cutting out the query portion (finding the root domain) by reusing the same base query each time (thus it will be cached).

          Actually, if I test a field with a cardinality of 1M, the performance drop is on the order of 12% for me too. The biggest contributor is most likely a higher cost to find the top N entries (the count array will have 1M entries) that is unrelated to the docvalues implementation.

          As far as replicating some of these results... I think most of the relevant details (including what exact queries look like) in SOLR-9599.
          Probably one of the simplest to replicate at the lucene level is a sorting test:

          http://localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW&fl=id&sort=s10_s%20desc,%20s100_s%20desc,%20s1000_s%20desc
          

          So basically, do a really inexpensive query that covers pretty much all of the index, and sorts by 3 fields (a field with a cardinality of 10, followed by a tiebreak with cardinality 100, followed by a tiebreak with cardinality 1000). That helps isolate sorting-by-docvalue performance. I quickly tested this by hand, and it was 50% slower (I just ran it multiple times and noted the lowest stable times).

          That's the wrong tradeoff, and we shouldn't let performance mess up our APIs that heavily.

          Subjectively, I would chose the other trade-off as it's our job to be fast. The previous API wasn't bad... it just needed help with sparse values.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - Hmm, in Lucene's nightly perf tests, the TermDateFacets got only a bit slower (~11%), not 40% slower. Yonik, can you give more details on your benchmark so others can run it? Amdahl's law? My tests are probably just isolating the docValues performance more. These are full-stack tests (on both sides?)... so it may be that TermDateFacets spends less of it's execution time actually retrieving docValues, and has more bottlenecks elsewhere. I'm also effectively cutting out the query portion (finding the root domain) by reusing the same base query each time (thus it will be cached). Actually, if I test a field with a cardinality of 1M, the performance drop is on the order of 12% for me too. The biggest contributor is most likely a higher cost to find the top N entries (the count array will have 1M entries) that is unrelated to the docvalues implementation. As far as replicating some of these results... I think most of the relevant details (including what exact queries look like) in SOLR-9599 . Probably one of the simplest to replicate at the lucene level is a sorting test: http: //localhost:8983/solr/collection1/query?q=*:*%20mydate_dt:NOW&fl=id&sort=s10_s%20desc,%20s100_s%20desc,%20s1000_s%20desc So basically, do a really inexpensive query that covers pretty much all of the index, and sorts by 3 fields (a field with a cardinality of 10, followed by a tiebreak with cardinality 100, followed by a tiebreak with cardinality 1000). That helps isolate sorting-by-docvalue performance. I quickly tested this by hand, and it was 50% slower (I just ran it multiple times and noted the lowest stable times). That's the wrong tradeoff, and we shouldn't let performance mess up our APIs that heavily. Subjectively, I would chose the other trade-off as it's our job to be fast. The previous API wasn't bad... it just needed help with sparse values.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -


          I had missed LUCENE-7462, thanks!

          Show
          yseeley@gmail.com Yonik Seeley added a comment - I had missed LUCENE-7462 , thanks!
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Sorting by numeric docvalues seems to have taken a greater hit (~45% for a low cardinality int field followed by another int field) than sorting by string for some reason.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - Sorting by numeric docvalues seems to have taken a greater hit (~45% for a low cardinality int field followed by another int field) than sorting by string for some reason.
          Hide
          mikemccand Michael McCandless added a comment -

          Well, your tests are also using synthetically generated data right? If you run performance tests with synthetic data, you draw synthetic conclusions.

          Can you test a real corpus instead? Maybe try faceting on e.g. city, state, country from geonames?

          I'll see if I can add a lower cardinality field from Wikipedia to Lucene's nightly benchmark.

          And while I appreciate your efforts to isolate doc values performance alone ("finding the root domain"), this is also a rather overly synthetic use case. Most queries involve non-trivial cost, and the overall impact to real world use cases is what matters here.

          Instead of "quickly testing things by hand" please do a more thorough test, discarding warmup iterations, running N JVMs (to account for hotstpot noise) with M iterations each (to account for other JVM noise) with diverse concurrent query types (to prevent hotspot from falsely over optimizing), etc.

          Separately, in scrutinizing the TermDateFacets charts in the nightly benchmark while digging here, I found a horrible bug in the benchmark code Caused by me on 5/25 and fixed in last night's run, but still the net before/after looks ~9% lower performance.

          Finally, please review the changes I had to do to Solr; maybe there's something silly there: you are the Solr expert here, try to be part of the solution

          Show
          mikemccand Michael McCandless added a comment - Well, your tests are also using synthetically generated data right? If you run performance tests with synthetic data, you draw synthetic conclusions. Can you test a real corpus instead? Maybe try faceting on e.g. city, state, country from geonames? I'll see if I can add a lower cardinality field from Wikipedia to Lucene's nightly benchmark. And while I appreciate your efforts to isolate doc values performance alone ("finding the root domain"), this is also a rather overly synthetic use case. Most queries involve non-trivial cost, and the overall impact to real world use cases is what matters here. Instead of "quickly testing things by hand" please do a more thorough test, discarding warmup iterations, running N JVMs (to account for hotstpot noise) with M iterations each (to account for other JVM noise) with diverse concurrent query types (to prevent hotspot from falsely over optimizing), etc. Separately, in scrutinizing the TermDateFacets charts in the nightly benchmark while digging here, I found a horrible bug in the benchmark code Caused by me on 5/25 and fixed in last night's run, but still the net before/after looks ~9% lower performance. Finally, please review the changes I had to do to Solr; maybe there's something silly there: you are the Solr expert here, try to be part of the solution
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Well, your tests are also using synthetically generated data right? If you run performance tests with synthetic data, you draw synthetic conclusions.

          Testing with real requests is what users should do with their specific requests. We have no single set of such typical requests... we have too many users with too many use cases.
          Generalized synthetic conclusions can be superior to a single "real world" use case that fails to cover enough scenarios that real users will encounter. At first blush, it doesn't look like the lucenebench tests cover sorting and faceting that well.

          And while I appreciate your efforts to isolate doc values performance alone ("finding the root domain"), this is also a rather overly synthetic use case. Most queries involve non-trivial cost, and the overall impact to real world use cases is what matters here.

          I disagree. If one is measuring performance of a faceting change, then isolate it. Then you can say "this change improved faceting performance on large cardinality fields by up to 50%".
          If a request is doing other expensive stuff, of course the overall implact will be smaller. One can make the impact arbitrarily small by adding other more expensive stuff to the request.

          Also, some users out there will experience an impact of that magnitude. We really don't have a single "typical" real-world use case... we have too many users trying to do too many crazy things. Everything that might be considered a corner case is often represented by real users who depend on that performance. For example, I've seen plenty of users who try to facet on dozens of fields per request.

          Instead of "quickly testing things by hand"

          A quick test by hand is still more informative than having no information at all. The accuracy may be lower, but when I see changes of the size I saw, I know that it needs further investigation!
          For example, I tested function queries (ValueSource) and sorting by multiple docvalue fields. Are either of these things tested at all in https://home.apache.org/~mikemccand/lucenebench/ ? The test names suggest that they are not, but it's hard to tell.
          And wrt to the by-hand sorting test, I did follow it up with a more thorough test:
          https://issues.apache.org/jira/browse/SOLR-9599?focusedCommentId=15584223&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15584223

          please do a more thorough test, discarding warmup iterations, running N JVMs (to account for hotstpot noise) with M iterations each (to account for other JVM noise) with diverse concurrent query types (to prevent hotspot from falsely over optimizing), etc.

          Yes, I did all that.
          Running diverse fields in the same JVM run is esp important to prevent hotspot from over-optimizing for a single field cardinality (since different cardinalities have different docvalues encodings).

          How many different numeric fields are concurrently sorted on for https://home.apache.org/~mikemccand/lucenebench/ ?
          The names suggest just one: " TermQuery (date/time sort)"
          If that is actually the case, then you're in danger of hotspot over-specializing for that single field/cardinality.

          try to be part of the solution

          That's an unnecessary personal dig.

          • I've already put in a lot of effort into benchmarking this, only to have it dismissed with hand waves, for cases that may not even be covered (or may be under stated) by your own benchmarks.
          • I fully intend to dig into the solr side, but I was waiting until the API stabilizes (LUCENE-7462)
          • I pointed at specific examples that reside entirely in lucene code (the sorting examples)
          Show
          yseeley@gmail.com Yonik Seeley added a comment - Well, your tests are also using synthetically generated data right? If you run performance tests with synthetic data, you draw synthetic conclusions. Testing with real requests is what users should do with their specific requests. We have no single set of such typical requests... we have too many users with too many use cases. Generalized synthetic conclusions can be superior to a single "real world" use case that fails to cover enough scenarios that real users will encounter. At first blush, it doesn't look like the lucenebench tests cover sorting and faceting that well. And while I appreciate your efforts to isolate doc values performance alone ("finding the root domain"), this is also a rather overly synthetic use case. Most queries involve non-trivial cost, and the overall impact to real world use cases is what matters here. I disagree. If one is measuring performance of a faceting change, then isolate it. Then you can say "this change improved faceting performance on large cardinality fields by up to 50%". If a request is doing other expensive stuff, of course the overall implact will be smaller. One can make the impact arbitrarily small by adding other more expensive stuff to the request. Also, some users out there will experience an impact of that magnitude. We really don't have a single "typical" real-world use case... we have too many users trying to do too many crazy things. Everything that might be considered a corner case is often represented by real users who depend on that performance. For example, I've seen plenty of users who try to facet on dozens of fields per request . Instead of "quickly testing things by hand" A quick test by hand is still more informative than having no information at all. The accuracy may be lower, but when I see changes of the size I saw, I know that it needs further investigation! For example, I tested function queries (ValueSource) and sorting by multiple docvalue fields. Are either of these things tested at all in https://home.apache.org/~mikemccand/lucenebench/ ? The test names suggest that they are not, but it's hard to tell. And wrt to the by-hand sorting test, I did follow it up with a more thorough test: https://issues.apache.org/jira/browse/SOLR-9599?focusedCommentId=15584223&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15584223 please do a more thorough test, discarding warmup iterations, running N JVMs (to account for hotstpot noise) with M iterations each (to account for other JVM noise) with diverse concurrent query types (to prevent hotspot from falsely over optimizing), etc. Yes, I did all that. Running diverse fields in the same JVM run is esp important to prevent hotspot from over-optimizing for a single field cardinality (since different cardinalities have different docvalues encodings). How many different numeric fields are concurrently sorted on for https://home.apache.org/~mikemccand/lucenebench/ ? The names suggest just one: " TermQuery (date/time sort)" If that is actually the case, then you're in danger of hotspot over-specializing for that single field/cardinality. try to be part of the solution That's an unnecessary personal dig. I've already put in a lot of effort into benchmarking this, only to have it dismissed with hand waves, for cases that may not even be covered (or may be under stated) by your own benchmarks. I fully intend to dig into the solr side, but I was waiting until the API stabilizes ( LUCENE-7462 ) I pointed at specific examples that reside entirely in lucene code (the sorting examples)
          Hide
          mikemccand Michael McCandless added a comment -

          At first blush, it doesn't look like the lucenebench tests cover sorting and faceting that well.

          For example, I tested function queries (ValueSource) and sorting by multiple docvalue fields. Are either of these things tested at all in https://home.apache.org/~mikemccand/lucenebench/ ?

          Running diverse fields in the same JVM run is esp important to prevent hotspot from over-optimizing for a single field cardinality (since different cardinalities have different docvalues encodings).

          How many different numeric fields are concurrently sorted on for https://home.apache.org/~mikemccand/lucenebench/ ?
          The names suggest just one: " TermQuery (date/time sort)"
          If that is actually the case, then you're in danger of hotspot over-specializing for that single field/cardinality.

          These are all good points, all things that I would like to improve
          about Lucene's nightly benchmarks
          (https://home.apache.org/~mikemccand/lucenebench/). Patches welcome

          I'll try to add some low cardinality faceting/sorting coverage, maybe
          using month name and day-of-the-year from the last modified date.

          The nightly Wikipedia benchmark facets on Date field as a hierarchy
          (year/month/day), and sorts on "last modified" (seconds resolution I
          think) and title.

          I've also long wanted to add highlighters...

          A quick test by hand is still more informative than having no information at all.

          I disagree: it's reckless to run an overly synthetic benchmark and
          then present the results as if they mean we should make poor API
          tradeoffs.

          If one is measuring performance of a faceting change, then isolate it.

          In the ideal world, yes, but this is notoriously problematic to do
          with java: hotspot, GC, etc. will all behave very differently if you
          are testing a very narrow part of the code.

          That's an unnecessary personal dig.
          I've already put in a lot of effort into benchmarking this, only to have it dismissed with hand waves, for cases that may not even be covered (or may be under stated) by your own benchmarks.
          I fully intend to dig into the solr side, but I was waiting until the API stabilizes (LUCENE-7462)
          I pointed at specific examples that reside entirely in lucene code (the sorting examples)

          My point is that running synthetic benchmarks and mis-representing
          them as "meaningful" is borderline reckless, and certainly nowhere
          near as helpful as, say, improving our default codec, profiling and
          removing slow spots, removing extra legacy wrappers, etc. Those are
          more positive ways to move our project forward.

          Perhaps you feel you have put in a lot of effort here, but from where
          I stand I see lots of complaining about how things got slower and little
          effort to actually improve the sources. This issue alone was a
          tremendous amount of slogging for me, and I had to switch Solr over
          without fully understanding its sources: you or other Solr experts
          could have stepped in to help me then.

          But why not do that now? I.e. review my Solr changes or function
          queries, etc.? I could easily have done something silly: it was just
          a "rote" cutover to the iterator API.

          I think we could nicely optimize the browse only case, by just using
          nextDoc to step through all doc values for a given field. Does Solr
          do that today?

          Why not test the patch on LUCENE-7462 to see if that API change helps?

          I am not disagreeing that DV access got slower: the Lucene nightly
          benchmarks also show that.

          Yet look at sort-by-title: at first it got slower, on initial cutover
          to iterators, but then thanks to Adrien Grand (thank you Adrien!), it's
          now faster than it was before:
          https://home.apache.org/~mikemccand/lucenebench/TermTitleSort.html

          With more iterations I expect we can do the same thing for the other
          dense cases. An iteration-only API means we can do all sorts of nice
          compression improvements not possible with the random access API, we
          don't need per-lookup bounds checks, etc. We should adopt from the
          many things we do to compress postings, which have been iterators only
          forever. And it means the sparse case, as a happy side effect,
          get to improve too.

          This could lead to a point in the future where the dense cases perform
          better than they did with random access API, like sort-by-title does
          already. We've only just begun down this path, and in just a few
          weeks Adrien Grand has already made big gains.

          Show
          mikemccand Michael McCandless added a comment - At first blush, it doesn't look like the lucenebench tests cover sorting and faceting that well. For example, I tested function queries (ValueSource) and sorting by multiple docvalue fields. Are either of these things tested at all in https://home.apache.org/~mikemccand/lucenebench/ ? Running diverse fields in the same JVM run is esp important to prevent hotspot from over-optimizing for a single field cardinality (since different cardinalities have different docvalues encodings). How many different numeric fields are concurrently sorted on for https://home.apache.org/~mikemccand/lucenebench/ ? The names suggest just one: " TermQuery (date/time sort)" If that is actually the case, then you're in danger of hotspot over-specializing for that single field/cardinality. These are all good points, all things that I would like to improve about Lucene's nightly benchmarks ( https://home.apache.org/~mikemccand/lucenebench/ ). Patches welcome I'll try to add some low cardinality faceting/sorting coverage, maybe using month name and day-of-the-year from the last modified date. The nightly Wikipedia benchmark facets on Date field as a hierarchy (year/month/day), and sorts on "last modified" (seconds resolution I think) and title. I've also long wanted to add highlighters... A quick test by hand is still more informative than having no information at all. I disagree: it's reckless to run an overly synthetic benchmark and then present the results as if they mean we should make poor API tradeoffs. If one is measuring performance of a faceting change, then isolate it. In the ideal world, yes, but this is notoriously problematic to do with java: hotspot, GC, etc. will all behave very differently if you are testing a very narrow part of the code. That's an unnecessary personal dig. I've already put in a lot of effort into benchmarking this, only to have it dismissed with hand waves, for cases that may not even be covered (or may be under stated) by your own benchmarks. I fully intend to dig into the solr side, but I was waiting until the API stabilizes ( LUCENE-7462 ) I pointed at specific examples that reside entirely in lucene code (the sorting examples) My point is that running synthetic benchmarks and mis-representing them as "meaningful" is borderline reckless, and certainly nowhere near as helpful as, say, improving our default codec, profiling and removing slow spots, removing extra legacy wrappers, etc. Those are more positive ways to move our project forward. Perhaps you feel you have put in a lot of effort here, but from where I stand I see lots of complaining about how things got slower and little effort to actually improve the sources. This issue alone was a tremendous amount of slogging for me, and I had to switch Solr over without fully understanding its sources: you or other Solr experts could have stepped in to help me then. But why not do that now? I.e. review my Solr changes or function queries, etc.? I could easily have done something silly: it was just a "rote" cutover to the iterator API. I think we could nicely optimize the browse only case, by just using nextDoc to step through all doc values for a given field. Does Solr do that today? Why not test the patch on LUCENE-7462 to see if that API change helps? I am not disagreeing that DV access got slower: the Lucene nightly benchmarks also show that. Yet look at sort-by-title: at first it got slower, on initial cutover to iterators, but then thanks to Adrien Grand (thank you Adrien!), it's now faster than it was before: https://home.apache.org/~mikemccand/lucenebench/TermTitleSort.html With more iterations I expect we can do the same thing for the other dense cases. An iteration-only API means we can do all sorts of nice compression improvements not possible with the random access API, we don't need per-lookup bounds checks, etc. We should adopt from the many things we do to compress postings, which have been iterators only forever. And it means the sparse case, as a happy side effect, get to improve too. This could lead to a point in the future where the dense cases perform better than they did with random access API, like sort-by-title does already. We've only just begun down this path, and in just a few weeks Adrien Grand has already made big gains.
          Hide
          dsmiley David Smiley added a comment -

          I wouldn't dare suggest to another committer how they should spend their time; it's entirely their prerogative. That's crossing a line; please stop! I think we should value all technical input, even if it's bad news (e.g. something got slower). Building/running a benchmark is being helpful. I understand if you don't like the benchmark in particular (I'm not going to argue it's a particularly good or bad one) but it's being helpful and it takes time to do these things. I'd be depressed right now if I were in Yonik's shoes; but hey that's me and we need emotions of steel around here to survive.

          Show
          dsmiley David Smiley added a comment - I wouldn't dare suggest to another committer how they should spend their time; it's entirely their prerogative. That's crossing a line; please stop! I think we should value all technical input, even if it's bad news (e.g. something got slower). Building/running a benchmark is being helpful. I understand if you don't like the benchmark in particular (I'm not going to argue it's a particularly good or bad one) but it's being helpful and it takes time to do these things. I'd be depressed right now if I were in Yonik's shoes; but hey that's me and we need emotions of steel around here to survive.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I think this is progressing in a good way. The initial work was done in master and not backported 6x. The initial work had a performance impact and it's been noted. Now it's time to work on improving performance. I'll be happy to help out with the performance issues. As long we don't have a need to rush out 7.0 then we have some time to improve the performance.

          Show
          joel.bernstein Joel Bernstein added a comment - I think this is progressing in a good way. The initial work was done in master and not backported 6x. The initial work had a performance impact and it's been noted. Now it's time to work on improving performance. I'll be happy to help out with the performance issues. As long we don't have a need to rush out 7.0 then we have some time to improve the performance.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          > A quick test by hand is still more informative than having no information at all.

          I disagree: it's reckless to run an overly synthetic benchmark and then present the results as if they mean we should make poor API tradeoffs.

          When I did a quick test by hand, I always disclosed that. It's a starting point, not an ending point.
          And even homogenous tests (that are prone to hotspot overspecialization) are a useful datapoint, if you know what they are.
          Some users will have exactly those types of requests - very homogeneous.

          My point is that running synthetic benchmarks and mis-representing them as "meaningful" is borderline reckless

          The implication being that you judge they are not meaningful? Wow.

          You seemed to admit that the lucene benchmarks don't even cover some of these cases (or don't cover them adequately).

          • There is no single authoritative benchmark, and it's misleading to suggest there is (that somehow represents the true performance for users)
          • The lucene benchmarks are also synthetic to a degree (although based off of real data). For example, the query cache is disabled. Why? I assume to better isolate what is being tested.
          • More realistic tests are always nice to verify that nothing was messed up... but a system will always have a bottleneck. The question is which bottleneck are you effectively testing?
          • More tests are better. If others have the time/ability, they should run their own!

          [...] nowhere near as helpful as, say, improving our default codec, profiling and removing slow spots, removing extra legacy wrappers, etc. Those are more positive ways to move our project forward.

          The first step I'd take would be to try and realistically isolate and quantify the performance of what I was trying to change anyway. I did that starting off with Solr faceting tests (lucene benchmarks don't test that).

          I will get around to trying and improve things.. in the meantime putting out the information I did have is better than hiding it. Take it for what it is.
          If you choose to just dismiss it as meaningless... well, I guess we'll have to agree to disagree.

          Show
          yseeley@gmail.com Yonik Seeley added a comment - > A quick test by hand is still more informative than having no information at all. I disagree: it's reckless to run an overly synthetic benchmark and then present the results as if they mean we should make poor API tradeoffs. When I did a quick test by hand, I always disclosed that. It's a starting point, not an ending point. And even homogenous tests (that are prone to hotspot overspecialization) are a useful datapoint, if you know what they are. Some users will have exactly those types of requests - very homogeneous. My point is that running synthetic benchmarks and mis-representing them as "meaningful" is borderline reckless The implication being that you judge they are not meaningful? Wow. You seemed to admit that the lucene benchmarks don't even cover some of these cases (or don't cover them adequately). There is no single authoritative benchmark, and it's misleading to suggest there is (that somehow represents the true performance for users) The lucene benchmarks are also synthetic to a degree (although based off of real data). For example, the query cache is disabled. Why? I assume to better isolate what is being tested. More realistic tests are always nice to verify that nothing was messed up... but a system will always have a bottleneck. The question is which bottleneck are you effectively testing ? More tests are better. If others have the time/ability, they should run their own! [...] nowhere near as helpful as, say, improving our default codec, profiling and removing slow spots, removing extra legacy wrappers, etc. Those are more positive ways to move our project forward. The first step I'd take would be to try and realistically isolate and quantify the performance of what I was trying to change anyway. I did that starting off with Solr faceting tests (lucene benchmarks don't test that). I will get around to trying and improve things.. in the meantime putting out the information I did have is better than hiding it. Take it for what it is. If you choose to just dismiss it as meaningless... well, I guess we'll have to agree to disagree.
          Hide
          mikemccand Michael McCandless added a comment -

          I think this is progressing in a good way. The initial work was done in master and not backported 6x. The initial work had a performance impact and it's been noted. Now it's time to work on improving performance. I'll be happy to help out with the performance issues. As long we don't have a need to rush out 7.0 then we have some time to improve the performance.

          Thank you Joel Bernstein!

          Show
          mikemccand Michael McCandless added a comment - I think this is progressing in a good way. The initial work was done in master and not backported 6x. The initial work had a performance impact and it's been noted. Now it's time to work on improving performance. I'll be happy to help out with the performance issues. As long we don't have a need to rush out 7.0 then we have some time to improve the performance. Thank you Joel Bernstein !
          Hide
          otis Otis Gospodnetic added a comment -

          I had a quick look at Yonik Seeley's SOLR-9599 and then at Adrien Grand's patch in LUCENE-7462 that makes the search-time work less expensive. Last comment from Yonik reporting faceting regression in Solr was from October 18. Adrien't patch was committed on October 24. Maybe things are working better for Solr now?

          If not, in interest of moving forward, what do people think about Yonik's suggestion:

          Perhaps we should have both a random access API as well as an iterator API?

          ?

          Show
          otis Otis Gospodnetic added a comment - I had a quick look at Yonik Seeley 's SOLR-9599 and then at Adrien Grand 's patch in LUCENE-7462 that makes the search-time work less expensive. Last comment from Yonik reporting faceting regression in Solr was from October 18. Adrien't patch was committed on October 24. Maybe things are working better for Solr now? If not, in interest of moving forward, what do people think about Yonik's suggestion: Perhaps we should have both a random access API as well as an iterator API? ?
          Hide
          mikemccand Michael McCandless added a comment -

          I don't think having to support two wildly different apis makes sense: that would be the worst of both worlds, because then the codec couldn't optimize to either, and we'd have bifurcated code all over Lucene, sometimes using one API, sometimes using another.

          Show
          mikemccand Michael McCandless added a comment - I don't think having to support two wildly different apis makes sense: that would be the worst of both worlds, because then the codec couldn't optimize to either, and we'd have bifurcated code all over Lucene, sometimes using one API, sometimes using another.
          Hide
          mkhludnev Mikhail Khludnev added a comment -
          TestBlockJoinSelector.testSortedSelector()
            final BitSet parents = new FixedBitSet(20);
          ...
              parents.set(15);
              parents.set(19);
          
              final BitSet children = new FixedBitSet(20);
          ..
              children.set(12);
              children.set(17);
          
              final int[] ords = new int[20];
              Arrays.fill(ords, -1);
          ...
              ords[12] = 10;
              ords[18] = 10;
          
              final SortedDocValues mins = BlockJoinSelector.wrap(DocValues.singleton(new CannedSortedDocValues(ords)), BlockJoinSelector.Type.MIN, parents, children);
          ...
              assertEquals(15, mins.nextDoc());
              assertEquals(10, mins.ordValue());
              assertEquals(19, mins.nextDoc()); // <---- why??? 
              assertEquals(10, mins.ordValue());
          ...
          

          19th parent has only 17th kid and value is assigned only on 18th.
          Michael McCandless, shouldn't it assert NO_MORE_DOCS here like it's done with numerics?

          Show
          mkhludnev Mikhail Khludnev added a comment - TestBlockJoinSelector.testSortedSelector() final BitSet parents = new FixedBitSet(20); ... parents.set(15); parents.set(19); final BitSet children = new FixedBitSet(20); .. children.set(12); children.set(17); final int [] ords = new int [20]; Arrays.fill(ords, -1); ... ords[12] = 10; ords[18] = 10; final SortedDocValues mins = BlockJoinSelector.wrap(DocValues.singleton( new CannedSortedDocValues(ords)), BlockJoinSelector.Type.MIN, parents, children); ... assertEquals(15, mins.nextDoc()); assertEquals(10, mins.ordValue()); assertEquals(19, mins.nextDoc()); // <---- why??? assertEquals(10, mins.ordValue()); ... 19th parent has only 17th kid and value is assigned only on 18th. Michael McCandless , shouldn't it assert NO_MORE_DOCS here like it's done with numerics?
          Hide
          jpountz Adrien Grand added a comment -

          Agreed the sorted selector is buggy. It seems to ignore cases when none of the matching children in a block have a value. It is also a pity that we require a BitSet for the children while we only need forward access, we should make it a DocIdSetIterator, which would have the nice side-effect that we could easily combine the doc values iterator and the child filter using a ConjunctionDISI to efficiently only iterate over child doc ids that both have a value and match the child filter.

          Show
          jpountz Adrien Grand added a comment - Agreed the sorted selector is buggy. It seems to ignore cases when none of the matching children in a block have a value. It is also a pity that we require a BitSet for the children while we only need forward access, we should make it a DocIdSetIterator, which would have the nice side-effect that we could easily combine the doc values iterator and the child filter using a ConjunctionDISI to efficiently only iterate over child doc ids that both have a value and match the child filter.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          spawned LUCENE-7871

          Show
          mkhludnev Mikhail Khludnev added a comment - spawned LUCENE-7871

            People

            • Assignee:
              mikemccand Michael McCandless
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development