Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.9
    • Fix Version/s: None
    • Component/s: modules/spatial
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      The attached patch is a large refactoring of the spatial search contrib. The primary contribution is the creation of the ThreadedDistanceFilter, which uses an ExecutorService to filter the documents in multiple threads. As a result of doing the filtering in multiple threads, the time taken to filter 1.2 million documents has been reduced from nearly 3s, to between 500-800ms.

      As part of this work, the DistanceQueryBuilder has been replaced by the SpatialFilter, a Lucene Filter, some unused functionality has been removed, and the package hierarchy has changed. Consequently this patch breaks backwards compatibility with the existing spatial search contrib.

      Also during the process of making these changes, abstractions have been added so that the one implementation of the ThreadedDistanceFilter can work with lat/long and geohash data formats, and so that precise but costly arc distance calculations can be replaced by less precise but much more efficient flat plane calculations if needed.

      This patch will be used in an upcoming patch for Solr which will improve Solr's support for spatial search.

        Issue Links

          Activity

          Chris Male created issue -
          Chris Male made changes -
          Field Original Value New Value
          Attachment LUCENE-1732_multi_threaded_spatial_search.patch [ 12412493 ]
          Hide
          Uwe Schindler added a comment - - edited

          As it breaks backwards compatibility, could you use o.a.l.util.NumericUtils instead of NumberUtils and remove the class? Using this new class, all numeric values use the same encoding coming from Lucene core. This is part of a new approach for numeric range queries (see also SOLR-940), but the number format can also be used for Spatial's Tier encoding.

          I could then close LUCENE-1505.

          Show
          Uwe Schindler added a comment - - edited As it breaks backwards compatibility, could you use o.a.l.util.NumericUtils instead of NumberUtils and remove the class? Using this new class, all numeric values use the same encoding coming from Lucene core. This is part of a new approach for numeric range queries (see also SOLR-940 ), but the number format can also be used for Spatial's Tier encoding. I could then close LUCENE-1505 .
          Hide
          Chris Male added a comment -

          Unfortunately Solr is not using a version of Lucene which has the NumericUtils class at this point and I would like for this patch to be usable with Solr. When Solr does update to a version of Lucene which does include NumericUtils, I will update my patch to use the class.

          Show
          Chris Male added a comment - Unfortunately Solr is not using a version of Lucene which has the NumericUtils class at this point and I would like for this patch to be usable with Solr. When Solr does update to a version of Lucene which does include NumericUtils, I will update my patch to use the class.
          Hide
          Uwe Schindler added a comment -

          This is only a couple of days until this is part of Solr trunk. This is a case about Lucene 2.9, so it should use Lucene 2.9 things. This contrib may be part of Solr some time in the future but until then the Lucene JARs will also be updated.

          I only mention this, as I will do the change to NumericUtils in very near future (when I have time to do it). NumericUtils has also the important advantage, that it is natively supported by Lucene's FieldCache (you can do e.g. FieldCache.getFloats() on such a field), so calculations on such fields could also be done using the FieldCache.

          The Lucene people have no problem with changing the API nor the index format, as this was not yet released, so there is no backwards problem (only for people already using another version of LocalLucene).

          Uwe

          Show
          Uwe Schindler added a comment - This is only a couple of days until this is part of Solr trunk. This is a case about Lucene 2.9, so it should use Lucene 2.9 things. This contrib may be part of Solr some time in the future but until then the Lucene JARs will also be updated. I only mention this, as I will do the change to NumericUtils in very near future (when I have time to do it). NumericUtils has also the important advantage, that it is natively supported by Lucene's FieldCache (you can do e.g. FieldCache.getFloats() on such a field), so calculations on such fields could also be done using the FieldCache. The Lucene people have no problem with changing the API nor the index format, as this was not yet released, so there is no backwards problem (only for people already using another version of LocalLucene). Uwe
          Chris Male made changes -
          Link This issue is part of LUCENE-2152 [ LUCENE-2152 ]
          Hide
          David Smiley added a comment -

          If I have a machine with say four CPU cores also running Solr with four cores (a distributed – i.e. sharded index), would it be fair to say that the optimization presented in this issue is of no use?

          Show
          David Smiley added a comment - If I have a machine with say four CPU cores also running Solr with four cores (a distributed – i.e. sharded index), would it be fair to say that the optimization presented in this issue is of no use?
          Hide
          Chris Male added a comment -

          Closing along with LUCENE-2139

          Show
          Chris Male added a comment - Closing along with LUCENE-2139
          Chris Male made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Assignee Chris Male [ cmale ]
          Resolution Won't Fix [ 2 ]
          Mark Thomas made changes -
          Workflow jira [ 12467462 ] Default workflow, editable Closed status [ 12564009 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12564009 ] jira [ 12585482 ]

            People

            • Assignee:
              Chris Male
              Reporter:
              Chris Male
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development