Solr
  1. Solr
  2. SOLR-2833

GeoNames webservice Update Processor

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: spatial

      Description

      A simple update processor for querying the GeoNames.org webservice. This should probably go in its own contrib module, where other geonames stuff by Mattmann & co could also go.

        Activity

        Hide
        Chris A. Mattmann added a comment -

        Just to comment back on this too:

        My current processor talks directly to Geonames.org, with caching of requests. This mode is good for development and testing, but for production I imagine that the backend could be configurable to either geonames.org webservice or local solr core. Comments?

        Cool yah, that's a good idea. The thing is, the easiest way for us was just to take geonames dumps (text files which we have on a bunch of our projects) and then as a first step index them in Solr creating our "geonames-aware" index. Then, this can be used during qparsing to allow those types of queries. I didn't want to go live to geonames.org b/c the site bandwidth might be an issue, and sometimes also we're working behind firewalls and stuff.

        Show
        Chris A. Mattmann added a comment - Just to comment back on this too: My current processor talks directly to Geonames.org, with caching of requests. This mode is good for development and testing, but for production I imagine that the backend could be configurable to either geonames.org webservice or local solr core. Comments? Cool yah, that's a good idea. The thing is, the easiest way for us was just to take geonames dumps (text files which we have on a bunch of our projects) and then as a first step index them in Solr creating our "geonames-aware" index. Then, this can be used during qparsing to allow those types of queries. I didn't want to go live to geonames.org b/c the site bandwidth might be an issue, and sometimes also we're working behind firewalls and stuff.
        Hide
        Chris A. Mattmann added a comment -

        That README file BTW on the github account is just a cut/paste from a message I posted long ago to the mailing list:

        http://search-lucene.com/m/9Uikk28pgc82/Geonames+Spatial+&subj=+Spatial+Geonames+and+extension+to+Spatial+Solution+for+Solr

        Show
        Chris A. Mattmann added a comment - That README file BTW on the github account is just a cut/paste from a message I posted long ago to the mailing list: http://search-lucene.com/m/9Uikk28pgc82/Geonames+Spatial+&subj=+Spatial+Geonames+and+extension+to+Spatial+Solution+for+Solr
        Hide
        Chris A. Mattmann added a comment -

        Hi Jan:

        Check out: SOLR-2082 (we use it to post geonames dump files, that's the first step). The second step is the update request processor (SOLR-2073) that stores geonames dumps as documents in the index. Then, we can query based on the spatial q parser (SOLR-2075). This lets you do queries like:

           {!spatial ct=[city] s=[state] c=[country] d=[search radius]}search text
           e.g. {!spatial ct=Orlando s=FL c=US d=400}NASA
        

        (which in our example looked up articles about NASA and JPL stories within
        400 km (or miles) of Orlando, FL in the US).

        You can find the full documentation here:

        https://github.com/chrismattmann/solrcene

        Show
        Chris A. Mattmann added a comment - Hi Jan: Check out: SOLR-2082 (we use it to post geonames dump files, that's the first step). The second step is the update request processor ( SOLR-2073 ) that stores geonames dumps as documents in the index. Then, we can query based on the spatial q parser ( SOLR-2075 ). This lets you do queries like: {!spatial ct=[city] s=[state] c=[country] d=[search radius]}search text e.g. {!spatial ct=Orlando s=FL c=US d=400}NASA (which in our example looked up articles about NASA and JPL stories within 400 km (or miles) of Orlando, FL in the US). You can find the full documentation here: https://github.com/chrismattmann/solrcene
        Hide
        Jan Høydahl added a comment -

        I have not tried out your patches, but I understand that you index the geonames dumps locally. I did not see any UpdateProcessor which queries this local index to actually do geocoding of addresses??

        My current processor talks directly to Geonames.org, with caching of requests. This mode is good for development and testing, but for production I imagine that the backend could be configurable to either geonames.org webservice or local solr core. Comments?

        Show
        Jan Høydahl added a comment - I have not tried out your patches, but I understand that you index the geonames dumps locally. I did not see any UpdateProcessor which queries this local index to actually do geocoding of addresses?? My current processor talks directly to Geonames.org, with caching of requests. This mode is good for development and testing, but for production I imagine that the backend could be configurable to either geonames.org webservice or local solr core. Comments?
        Hide
        Chris A. Mattmann added a comment -

        Hi Jan: thanks for looking into this. Here's a full set of the issues that I worked on with my student, William:

        SOLR-2073 Geonames.org UpdateProcessor for Spatial
        SOLR-2074 GeoRSS ResponseWriter
        SOLR-2075 SpatialQParserPlugin and HostIP adaptor
        SOLR-2076 Spatial example schema updates
        SOLR-2077 Spatial example solconfig updates
        SOLR-2079 Expose HttpServletRequest object from SolrQueryRequest object
        SOLR-2081 BaseResponseWriter isStreamingDocs causes SingleResponseWriter.end
        to be called 2x
        SOLR-2082 Geopost.jar for loading geonames data

        They are already attached to JIRA and in patch form. I'd be very happy to work with you to contribute these. Thank you!

        (p.s. you might want to make some of these sub-tasks to this top-level task)

        Show
        Chris A. Mattmann added a comment - Hi Jan: thanks for looking into this. Here's a full set of the issues that I worked on with my student, William: SOLR-2073 Geonames.org UpdateProcessor for Spatial SOLR-2074 GeoRSS ResponseWriter SOLR-2075 SpatialQParserPlugin and HostIP adaptor SOLR-2076 Spatial example schema updates SOLR-2077 Spatial example solconfig updates SOLR-2079 Expose HttpServletRequest object from SolrQueryRequest object SOLR-2081 BaseResponseWriter isStreamingDocs causes SingleResponseWriter.end to be called 2x SOLR-2082 Geopost.jar for loading geonames data They are already attached to JIRA and in patch form. I'd be very happy to work with you to contribute these. Thank you! (p.s. you might want to make some of these sub-tasks to this top-level task)
        Hide
        Jan Høydahl added a comment -

        The first version will implement the Postal Code search servive, taking as input a country code together with postalCode and/or postalPlace and returns city name, lat, long and any other place names assocciated with the postal code.

        I suggest to start with this part and then incrementally add support for other geonames services.

        Show
        Jan Høydahl added a comment - The first version will implement the Postal Code search servive , taking as input a country code together with postalCode and/or postalPlace and returns city name, lat, long and any other place names assocciated with the postal code. I suggest to start with this part and then incrementally add support for other geonames services.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jan Høydahl
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development