Solr
  1. Solr
  2. SOLR-2155

Geospatial search using geohash prefixes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      NOTICE

      The status of this issue is a plugin for Solr 3.x located here: https://github.com/dsmiley/SOLR-2155. Look at the introductory readme and download the plugin .jar file. Lucene 4's new spatial module is largely based on this code. The Solr 4 glue for it should come very soon but as of this writing it's hosted temporarily at https://github.com/spatial4j. For more information on using SOLR-2155 with Solr 3, see http://wiki.apache.org/solr/SpatialSearch#SOLR-2155 This JIRA issue is closed because it won't be committed in its current form.

      There currently isn't a solution in Solr for doing geospatial filtering on documents that have a variable number of points. This scenario occurs when there is location extraction (i.e. via a "gazateer") occurring on free text. None, one, or many geospatial locations might be extracted from any given document and users want to limit their search results to those occurring in a user-specified area.

      I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash prefix based filter. A geohash refers to a lat-lon box on the earth. Each successive character added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the geohash) grid. The first step in this scheme is figuring out which geohash grid squares cover the user's search query. I've added various extra methods to GeoHashUtils (and added tests) to assist in this purpose. The next step is an actual Lucene Filter, GeoHashPrefixFilter, that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the index. Once a matching geohash grid is found, the points therein are compared against the user's query to see if it matches. I created an abstraction GeoShape extended by subclasses named PointDistance... and CartesianBox.... to support different queried shapes so that the filter need not care about these details.

      This work was presented at LuceneRevolution in Boston on October 8th.

      1. GeoHashPrefixFilter.patch
        69 kB
        David Smiley
      2. GeoHashPrefixFilter.patch
        49 kB
        David Smiley
      3. GeoHashPrefixFilter.patch
        37 kB
        David Smiley
      4. SOLR.2155.p3.patch
        10 kB
        Bill Bell
      5. SOLR.2155.p3tests.patch
        86 kB
        Bill Bell
      6. SOLR-2155_GeoHashPrefixFilter_with_sorting_no_poly.patch
        177 kB
        David Smiley
      7. Solr2155-1.0.2-project.zip
        95 kB
        David Smiley
      8. Solr2155-1.0.3-project.zip
        96 kB
        David Smiley
      9. Solr2155-1.0.4-project.zip
        136 kB
        David Smiley
      10. Solr2155-for-1.0.2-3.x-port.patch
        5 kB
        Mikhail Khludnev

        Issue Links

          Activity

          David Smiley created issue -
          David Smiley made changes -
          Field Original Value New Value
          Attachment GeoHashPrefixFilter.patch [ 12457039 ]
          Bill Bell made changes -
          Link This issue duplicates SOLR-2154 [ SOLR-2154 ]
          David Smiley made changes -
          Attachment GeoHashPrefixFilter.patch [ 12468694 ]
          David Smiley made changes -
          Attachment GeoHashPrefixFilter.patch [ 12469714 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p2.patch [ 12470663 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p2.patch [ 12470887 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p2.patch [ 12470663 ]
          Bill Bell made changes -
          Link This issue is related to SOLR-2345 [ SOLR-2345 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p3.patch [ 12471059 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p2.patch [ 12470887 ]
          Bill Bell made changes -
          Attachment SOLR.2155.p3tests.patch [ 12471117 ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          David Smiley made changes -
          David Smiley made changes -
          Attachment Solr2155-1.0.2-project.zip [ 12497003 ]
          Mikhail Khludnev made changes -
          Attachment Solr2155-for-1.0.2-3.x-port.patch [ 12497774 ]
          David Smiley made changes -
          Attachment Solr2155-1.0.3-project.zip [ 12497800 ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          David Smiley made changes -
          Link This issue is related to LUCENE-3795 [ LUCENE-3795 ]
          Harley Parks made changes -
          Comment [ I am missing the package solr2155.lucene.spatial.geometry.shape;

          ]
          Harley Parks made changes -
          Comment [ I used a fresh install of solr 3.4, Just in case our implementation had something to do with it, still not able to return the lat,long and only get the geohash.

          what's interesting is I can change the field's type between solr-2155's geohash & solr's geohash, the query returned format changes from geohash to lat,long without reindexing.

          Where in the code does the geohash value get returned in a query?

          Help. ]
          David Smiley made changes -
          Attachment Solr2155-1.0.4-project.zip [ 12519014 ]
          David Smiley made changes -
          Description There currently isn't a solution in Solr for doing geospatial filtering on documents that have a variable number of points. This scenario occurs when there is location extraction (i.e. via a "gazateer") occurring on free text. None, one, or many geospatial locations might be extracted from any given document and users want to limit their search results to those occurring in a user-specified area.

          I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash prefix based filter. A geohash refers to a lat-lon box on the earth. Each successive character added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the geohash) grid. The first step in this scheme is figuring out which geohash grid squares cover the user's search query. I've added various extra methods to GeoHashUtils (and added tests) to assist in this purpose. The next step is an actual Lucene Filter, GeoHashPrefixFilter, that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the index. Once a matching geohash grid is found, the points therein are compared against the user's query to see if it matches. I created an abstraction GeoShape extended by subclasses named PointDistance... and CartesianBox.... to support different queried shapes so that the filter need not care about these details.

          This work was presented at LuceneRevolution in Boston on October 8th.
          {panel:title=NOTICE} There is a .zip attached to the issue with a .jar you can drop in to Solr 3.x. Lucene 4's new spatial module is largely based on this code. The Solr 4 glue for it should come very soon but as of this writing it's hosted temporarily at Spatial4j.com. For more information on using SOLR-2155 with Solr 3, see http://wiki.apache.org/solr/SpatialSearch#SOLR-2155 This JIRA issue is closed because it won't be committed in its current form.
          {panel}

          There currently isn't a solution in Solr for doing geospatial filtering on documents that have a variable number of points. This scenario occurs when there is location extraction (i.e. via a "gazateer") occurring on free text. None, one, or many geospatial locations might be extracted from any given document and users want to limit their search results to those occurring in a user-specified area.

          I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash prefix based filter. A geohash refers to a lat-lon box on the earth. Each successive character added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the geohash) grid. The first step in this scheme is figuring out which geohash grid squares cover the user's search query. I've added various extra methods to GeoHashUtils (and added tests) to assist in this purpose. The next step is an actual Lucene Filter, GeoHashPrefixFilter, that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the index. Once a matching geohash grid is found, the points therein are compared against the user's query to see if it matches. I created an abstraction GeoShape extended by subclasses named PointDistance... and CartesianBox.... to support different queried shapes so that the filter need not care about these details.

          This work was presented at LuceneRevolution in Boston on October 8th.
          David Smiley made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Won't Fix [ 2 ]
          Bill Bell made changes -
          Link This issue blocks SOLR-3304 [ SOLR-3304 ]
          David Smiley made changes -
          Resolution Won't Fix [ 2 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Assignee David Smiley [ dsmiley ]
          David Smiley made changes -
          Description {panel:title=NOTICE} There is a .zip attached to the issue with a .jar you can drop in to Solr 3.x. Lucene 4's new spatial module is largely based on this code. The Solr 4 glue for it should come very soon but as of this writing it's hosted temporarily at Spatial4j.com. For more information on using SOLR-2155 with Solr 3, see http://wiki.apache.org/solr/SpatialSearch#SOLR-2155 This JIRA issue is closed because it won't be committed in its current form.
          {panel}

          There currently isn't a solution in Solr for doing geospatial filtering on documents that have a variable number of points. This scenario occurs when there is location extraction (i.e. via a "gazateer") occurring on free text. None, one, or many geospatial locations might be extracted from any given document and users want to limit their search results to those occurring in a user-specified area.

          I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash prefix based filter. A geohash refers to a lat-lon box on the earth. Each successive character added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the geohash) grid. The first step in this scheme is figuring out which geohash grid squares cover the user's search query. I've added various extra methods to GeoHashUtils (and added tests) to assist in this purpose. The next step is an actual Lucene Filter, GeoHashPrefixFilter, that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the index. Once a matching geohash grid is found, the points therein are compared against the user's query to see if it matches. I created an abstraction GeoShape extended by subclasses named PointDistance... and CartesianBox.... to support different queried shapes so that the filter need not care about these details.

          This work was presented at LuceneRevolution in Boston on October 8th.
          {panel:title=NOTICE} The status of this issue is a plugin for Solr 3.x located here: https://github.com/dsmiley/SOLR-2155. Look at the introductory readme and download the plugin .jar file. Lucene 4's new spatial module is largely based on this code. The Solr 4 glue for it should come very soon but as of this writing it's hosted temporarily at https://github.com/spatial4j. For more information on using SOLR-2155 with Solr 3, see http://wiki.apache.org/solr/SpatialSearch#SOLR-2155 This JIRA issue is closed because it won't be committed in its current form.
          {panel}

          There currently isn't a solution in Solr for doing geospatial filtering on documents that have a variable number of points. This scenario occurs when there is location extraction (i.e. via a "gazateer") occurring on free text. None, one, or many geospatial locations might be extracted from any given document and users want to limit their search results to those occurring in a user-specified area.

          I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash prefix based filter. A geohash refers to a lat-lon box on the earth. Each successive character added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the geohash) grid. The first step in this scheme is figuring out which geohash grid squares cover the user's search query. I've added various extra methods to GeoHashUtils (and added tests) to assist in this purpose. The next step is an actual Lucene Filter, GeoHashPrefixFilter, that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the index. Once a matching geohash grid is found, the points therein are compared against the user's query to see if it matches. I created an abstraction GeoShape extended by subclasses named PointDistance... and CartesianBox.... to support different queried shapes so that the filter need not care about these details.

          This work was presented at LuceneRevolution in Boston on October 8th.
          David Smiley made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Won't Fix [ 2 ]
          David Smiley made changes -
          Link This issue blocks SOLR-3304 [ SOLR-3304 ]
          David Smiley made changes -
          Link This issue is part of SOLR-3304 [ SOLR-3304 ]

            People

            • Assignee:
              David Smiley
              Reporter:
              David Smiley
            • Votes:
              21 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development