Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5
    • Component/s: spatial
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None
    • Flags:
      Patch

      Description

      The fastest and most efficient spatial field for point data in Lucene/Solr is LatLonPoint in Lucene's sandbox module. I'll include LatLonDocValuesField with this even though it's a separate class. LatLonPoint is based on the Points API, using a BKD index. It's multi-valued capable. LatLonDocValuesField is based on sorted numeric DocValues, and thus is also multi-valued capable (a big deal as the existing Solr ones either aren't or do poorly at it). Note that this feature is limited to a latitude/longitude spherical world model. And furthermore the precision is at about a centimeter – less precise than the other spatial fields but nonetheless plenty good for most applications. Last but not least, this capability natively supports polygons, albeit those that don't wrap the dateline or a pole.

      I propose a LatLonPointSpatialField which uses this. Patch & details forthcoming...

      This development was funded by the Harvard Center for Geographic Analysis as part of the HHypermap project

      1. SOLR_10039_LatLonPointSpatialField.patch
        24 kB
        David Smiley
      2. SOLR_10039_LatLonPointSpatialField.patch
        18 kB
        David Smiley
      3. SOLR_10039_LatLonPointSpatialField.patch
        13 kB
        David Smiley
      4. SOLR_10039_LatLonPointSpatialField.patch
        11 kB
        David Smiley

        Issue Links

          Activity

          Hide
          dsmiley David Smiley added a comment -

          The attached patch has most everything but it's not quite committable. I'm internally using a 6.4 based branch so the diff includes stuff that won't be needed for trunk or 6.5+.

          The field extends AbstractSpatialFieldType and thus inherits the functionality and integration with the rest of Solr spatial. The main TODOs are:

          • make indexed & docValues attributes configurable
          • integrate Polygon support.

          I would have liked to have introduced this embedded SpatialStrategy implementation to Lucene spatial-extras but I didn't think depending on sandbox was a good idea, at least not now, so I opted not to.

          Show
          dsmiley David Smiley added a comment - The attached patch has most everything but it's not quite committable. I'm internally using a 6.4 based branch so the diff includes stuff that won't be needed for trunk or 6.5+. The field extends AbstractSpatialFieldType and thus inherits the functionality and integration with the rest of Solr spatial. The main TODOs are: make indexed & docValues attributes configurable integrate Polygon support. I would have liked to have introduced this embedded SpatialStrategy implementation to Lucene spatial-extras but I didn't think depending on sandbox was a good idea, at least not now, so I opted not to.
          Hide
          dsmiley David Smiley added a comment -

          The patch includes some references to a HeatmapSpatialField that is erroneous for this patch as it's actually for another issue I'm working on.

          Show
          dsmiley David Smiley added a comment - The patch includes some references to a HeatmapSpatialField that is erroneous for this patch as it's actually for another issue I'm working on.
          Hide
          yseeley@gmail.com Yonik Seeley added a comment -

          Glad to see progress on this front!

          Regarding this:

          +      public SortField getSortField(boolean reverse) {
          +        //nocommit reverse?  throw exception if unsupported?
          +        return LatLonDocValuesField.newDistanceSort(fieldName, queryPoint.getY(), queryPoint.getX());
          +      }
          

          We should definitely throw an exception if it's not supported, but given that we should be able to have a value source representing the distance, we should be able to use that to easily construct a reverse sort?

          Show
          yseeley@gmail.com Yonik Seeley added a comment - Glad to see progress on this front! Regarding this: + public SortField getSortField( boolean reverse) { + //nocommit reverse? throw exception if unsupported? + return LatLonDocValuesField.newDistanceSort(fieldName, queryPoint.getY(), queryPoint.getX()); + } We should definitely throw an exception if it's not supported, but given that we should be able to have a value source representing the distance, we should be able to use that to easily construct a reverse sort?
          Hide
          dsmiley David Smiley added a comment -

          Great point; if reverse is true then I can merely call the superclass version

          I want to delay polygon support for another issue; there's actually a lot to make that work, and the functionality here is plenty good as-is.

          Show
          dsmiley David Smiley added a comment - Great point; if reverse is true then I can merely call the superclass version I want to delay polygon support for another issue; there's actually a lot to make that work, and the functionality here is plenty good as-is.
          Hide
          dsmiley David Smiley added a comment -

          New patch:

          • indexed & docValues are now properly configurable
          • tested geodist() with this field, and ascending/descending.

          I think it's ready, but I want to expand the patch to add this field type to our example configs in at least every file that already includes a spatial field, ordered just ahead of them so users are more likely to use this one.

          Query by polygon is left as a future TODO.

          Show
          dsmiley David Smiley added a comment - New patch: indexed & docValues are now properly configurable tested geodist() with this field, and ascending/descending. I think it's ready, but I want to expand the patch to add this field type to our example configs in at least every file that already includes a spatial field, ordered just ahead of them so users are more likely to use this one. Query by polygon is left as a future TODO.
          Hide
          dsmiley David Smiley added a comment -

          I might get a bit of time to share some performance numbers; although it's a bit of a forlorn conclusion that of course it's faster given the Lucene nightly benchmarks: https://home.apache.org/~mikemccand/geobench.html

          Show
          dsmiley David Smiley added a comment - I might get a bit of time to share some performance numbers; although it's a bit of a forlorn conclusion that of course it's faster given the Lucene nightly benchmarks: https://home.apache.org/~mikemccand/geobench.html
          Hide
          dsmiley David Smiley added a comment -

          Last patch; I think it's ready now. This replaces solr.LatLonType with solr.LatLonPointSpatialField in the solr/server/solr/configsets/ schemas.

          However it doesn't modify solr/example/ schemas... I wish those schemas were stripped down so that they were easier to maintain (separate issue). Thoughts Alexandre Rafalovitch?

          Show
          dsmiley David Smiley added a comment - Last patch; I think it's ready now. This replaces solr.LatLonType with solr.LatLonPointSpatialField in the solr/server/solr/configsets/ schemas. However it doesn't modify solr/example/ schemas... I wish those schemas were stripped down so that they were easier to maintain (separate issue). Thoughts Alexandre Rafalovitch ?
          Hide
          arafalov Alexandre Rafalovitch added a comment -

          I think we can just remove that field and related definitions from all the legacy examples. I don't see anything rely on them by definition.

          And yes, I think at least DIH examples should be stripped to absolute minimum. I was going to do that for Tika example in SOLR-9601. Is that the kind of things you were thinking about?

          Show
          arafalov Alexandre Rafalovitch added a comment - I think we can just remove that field and related definitions from all the legacy examples. I don't see anything rely on them by definition. And yes, I think at least DIH examples should be stripped to absolute minimum. I was going to do that for Tika example in SOLR-9601 . Is that the kind of things you were thinking about?
          Hide
          dsmiley David Smiley added a comment -

          Yes; thanks Alexandre.

          I'll commit this patch within a couple days or sooner if I get a +1.

          Show
          dsmiley David Smiley added a comment - Yes; thanks Alexandre. I'll commit this patch within a couple days or sooner if I get a +1.
          Hide
          dsmiley David Smiley added a comment -

          I've got another patch revision here. Essentially I noticed/forgot about Lucene's IndexOrDocValuesQuery which in turn made me also realize this field could support queries even if it has only docValues. So I added both. I also expanded the testing to include some variations.

          Show
          dsmiley David Smiley added a comment - I've got another patch revision here. Essentially I noticed/forgot about Lucene's IndexOrDocValuesQuery which in turn made me also realize this field could support queries even if it has only docValues. So I added both. I also expanded the testing to include some variations.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 182c20c4e55c39362f6d46d388eb2b8e0a9052e6 in lucene-solr's branch refs/heads/master from David Smiley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=182c20c ]

          SOLR-10039: New LatLonPointSpatialField

          Show
          jira-bot ASF subversion and git services added a comment - Commit 182c20c4e55c39362f6d46d388eb2b8e0a9052e6 in lucene-solr's branch refs/heads/master from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=182c20c ] SOLR-10039 : New LatLonPointSpatialField
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f171d181cb7a01c318b0a37e93897bf9f1fcf50f in lucene-solr's branch refs/heads/branch_6x from David Smiley
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f171d18 ]

          SOLR-10039: New LatLonPointSpatialField

          (cherry picked from commit 182c20c)

          Show
          jira-bot ASF subversion and git services added a comment - Commit f171d181cb7a01c318b0a37e93897bf9f1fcf50f in lucene-solr's branch refs/heads/branch_6x from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f171d18 ] SOLR-10039 : New LatLonPointSpatialField (cherry picked from commit 182c20c)
          Hide
          hossman Hoss Man added a comment -

          David Smiley: can you please update the ref guide with some guidance on this new field type?

          Replacing LatLonType with LatLonPointSpatialField in the list of field types is easy, but the way the "Spatial" page is written updating it to make meaningful comments about LatLonPointSpatialField (as compared to RPT) is a lot harder w/o some first hand knowledge about the merits...

          Show
          hossman Hoss Man added a comment - David Smiley : can you please update the ref guide with some guidance on this new field type? Replacing LatLonType with LatLonPointSpatialField in the list of field types is easy, but the way the "Spatial" page is written updating it to make meaningful comments about LatLonPointSpatialField (as compared to RPT) is a lot harder w/o some first hand knowledge about the merits... https://cwiki.apache.org/confluence/display/solr/Spatial+Search https://cwiki.apache.org/confluence/display/solr/Field+Types+Included+with+Solr
          Hide
          dsmiley David Smiley added a comment -

          David Smiley: can you please update the ref guide with some guidance on this new field type?

          Absolutely; I planned to do so tonight. I know I have to beat the clock before the ref guide RC sometime tomorrow.

          Show
          dsmiley David Smiley added a comment - David Smiley: can you please update the ref guide with some guidance on this new field type? Absolutely; I planned to do so tonight. I know I have to beat the clock before the ref guide RC sometime tomorrow.

            People

            • Assignee:
              dsmiley David Smiley
              Reporter:
              dsmiley David Smiley
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development