Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: spatial
    • Labels:
      None

      Description

      LUCENE-5648 introduced a date range index & search capability in the spatial module. This issue is for a corresponding Solr FieldType to be named "DateRangeField". LUCENE-5648 includes a parseCalendar(String) method that parses a superset of Solr's strict date format. It also parses partial dates (e.g.: 2014-10 has month specificity), and the trailing 'Z' is optional, and a leading +/- may be present (minus indicates BC era), and "*" means all-time. The proposed field type would use it to parse a string and also both ends of a range query, but furthermore it will also allow an arbitrary range query of the form <calspec> TO <calspec> such as:

      2000 TO 2014-05-21T10

      Which parses as the year 2000 thru 2014 May 21st 10am (GMT).
      I suggest this syntax because it is aligned with Lucene's range query syntax.

      1. SOLR-6103_more_tests.patch
        4 kB
        Varun Thacker
      2. SOLR-6103.patch
        10 kB
        David Smiley

        Issue Links

          Activity

          Hide
          David Smiley added a comment -

          It just occurred to me that

          * TO 2014

          ought to be supported but it doesn't work – I'll fix that in LUCENE-5648.

          Perhaps the range syntax should include matching '[' and ']'? It's only pertinent for indexing ranges; at query time you might as well use the normal range query syntax. One aspect I haven't considered is exclusive boundaries, but I think it's generally a non-issue because of the rounding this field supports.

          Note that LUCENE-5648 is still only v5/trunk for the moment.

          Show
          David Smiley added a comment - It just occurred to me that * TO 2014 ought to be supported but it doesn't work – I'll fix that in LUCENE-5648 . Perhaps the range syntax should include matching ' [' and '] '? It's only pertinent for indexing ranges; at query time you might as well use the normal range query syntax. One aspect I haven't considered is exclusive boundaries, but I think it's generally a non-issue because of the rounding this field supports. Note that LUCENE-5648 is still only v5/trunk for the moment.
          Hide
          David Smiley added a comment -

          I updated LUCENE-5608 with the date range parsing, and now I added the DateRangeField here with tests. Examples on how to index ranges are below. Note that they aren't necessarily explicit ranges, it can be implied by referring specifying a date instance to a desired granularity. It includes the same syntax Solr supports, though doesn't do "DateMath".

          [* TO *]
          2014-05-21T12:00:00.000Z
          [2000 TO 2014-05-21]
          

          By default, at search time the predicate is intersects, which means it'll match any overlap with an indexed date range. It can be specified with "op" as a local-param.

          q=dateRange:"2014-05-21"
          q={!field f=dateRange op=Contains v="[1999 TO 2001]"}
          

          I opted for this new "op" local-param instead of using Lucene-spatial's awkward SpatialArgsParser format which looks like "Intersects(foo)".

          Show
          David Smiley added a comment - I updated LUCENE-5608 with the date range parsing, and now I added the DateRangeField here with tests. Examples on how to index ranges are below. Note that they aren't necessarily explicit ranges, it can be implied by referring specifying a date instance to a desired granularity. It includes the same syntax Solr supports, though doesn't do "DateMath". [* TO *] 2014-05-21T12:00:00.000Z [2000 TO 2014-05-21] By default, at search time the predicate is intersects, which means it'll match any overlap with an indexed date range. It can be specified with "op" as a local-param. q=dateRange:"2014-05-21" q={!field f=dateRange op=Contains v="[1999 TO 2001]"} I opted for this new "op" local-param instead of using Lucene-spatial's awkward SpatialArgsParser format which looks like "Intersects(foo)".
          Hide
          Ryan McKinley added a comment -

          +1

          Show
          Ryan McKinley added a comment - +1
          Hide
          ASF subversion and git services added a comment -

          Commit 1600556 from David Smiley in branch 'dev/trunk'
          [ https://svn.apache.org/r1600556 ]

          SOLR-6103: Add QParser arg to AbstractSpatialFieldType.parseSpatialArgs(). Make getQueryFromSpatialArgs protected no private.

          Show
          ASF subversion and git services added a comment - Commit 1600556 from David Smiley in branch 'dev/trunk' [ https://svn.apache.org/r1600556 ] SOLR-6103 : Add QParser arg to AbstractSpatialFieldType.parseSpatialArgs(). Make getQueryFromSpatialArgs protected no private.
          Hide
          ASF subversion and git services added a comment -

          Commit 1600557 from David Smiley in branch 'dev/trunk'
          [ https://svn.apache.org/r1600557 ]

          SOLR-6103: DateRangeField

          Show
          ASF subversion and git services added a comment - Commit 1600557 from David Smiley in branch 'dev/trunk' [ https://svn.apache.org/r1600557 ] SOLR-6103 : DateRangeField
          Hide
          David Smiley added a comment -

          Committed to 5x for now; intend to move to 4x soon-ish.

          Please try it out folks!

          Faceting to come...

          Show
          David Smiley added a comment - Committed to 5x for now; intend to move to 4x soon-ish. Please try it out folks! Faceting to come...
          Hide
          Adrien Brault added a comment -

          David Smiley Are you still planning to get that change into 4.x ?

          It would allow us to solve a lot of problems that are way too hard to implement without the DateRangeField.

          Show
          Adrien Brault added a comment - David Smiley Are you still planning to get that change into 4.x ? It would allow us to solve a lot of problems that are way too hard to implement without the DateRangeField.
          Hide
          David Smiley added a comment -

          I am; I'm waiting to port some related API improvements once I figure out one last issue. Have you tried the feature on 5x? It's very easy to.

          Show
          David Smiley added a comment - I am; I'm waiting to port some related API improvements once I figure out one last issue. Have you tried the feature on 5x? It's very easy to.
          Hide
          Jack Krupansky added a comment -

          You might want to take a peek at the LucidWorks Search query parser support of date queries. It would be so nice to have comparable date support in Solr itself.

          It includes the ability to auto-expand a simple partial date/time term into a full range, as well as using partial date/time in explicit range queries.

          See:
          http://docs.lucidworks.com/display/lweug/Date+Queries

          Show
          Jack Krupansky added a comment - You might want to take a peek at the LucidWorks Search query parser support of date queries. It would be so nice to have comparable date support in Solr itself. It includes the ability to auto-expand a simple partial date/time term into a full range, as well as using partial date/time in explicit range queries. See: http://docs.lucidworks.com/display/lweug/Date+Queries
          Hide
          David Smiley added a comment -

          Thanks Jack; although I feel that's a separate issue. But at least this field does let you specify convenient prefixes of Solr's date syntax, and then get the range equivalent for that unit of time.

          Show
          David Smiley added a comment - Thanks Jack; although I feel that's a separate issue. But at least this field does let you specify convenient prefixes of Solr's date syntax, and then get the range equivalent for that unit of time.
          Hide
          Jack Krupansky added a comment - - edited

          One nuance is for the end of the range - [2010 TO 2012] should expand the starting date to the beginning of that period, but expand the ending date to the end of that period (2012-12-31T23:59:59.999Z). And [2010 TO 2012} would expand the ending date to the beginning (rather than the ending) of the period (2012-01-01T00:00:00Z), with the "exclusive" flag set as well.

          Show
          Jack Krupansky added a comment - - edited One nuance is for the end of the range - [2010 TO 2012] should expand the starting date to the beginning of that period, but expand the ending date to the end of that period (2012-12-31T23:59:59.999Z). And [2010 TO 2012} would expand the ending date to the beginning (rather than the ending) of the period (2012-01-01T00:00:00Z), with the "exclusive" flag set as well.
          Hide
          David Smiley added a comment - - edited

          I like that idea; it's only inclusive right now ']'.

          Show
          David Smiley added a comment - - edited I like that idea; it's only inclusive right now ']'.
          Hide
          Varun Thacker added a comment -

          Hi David,

          Cool feature!

          I wanted to add some tests when the 'dateRange' field has multiple ranges but wanted to clear out a doubt before I put up the patch -

          What is the difference when I search with

          [2000 TO 2014]

          instead of

          [2000-00 TO 2014-12]

          I can see that the Filter query being formed for the later has NRShape

          [1999-12 TO 2014]
          Show
          Varun Thacker added a comment - Hi David, Cool feature! I wanted to add some tests when the 'dateRange' field has multiple ranges but wanted to clear out a doubt before I put up the patch - What is the difference when I search with [2000 TO 2014] instead of [2000-00 TO 2014-12] I can see that the Filter query being formed for the later has NRShape [1999-12 TO 2014]
          Hide
          David Smiley added a comment -

          I'm glad you like it.

          2000-00 is wrong because the displayed and toString January is 01, not 00. It's true that internally 0 is used. Arguably this should throw an exception but I suppose the current behavior is not unexpected.

          [2000 TO 2014] and [2000-01 to 2014-12] should be equivalent. FYI there are 3 tests at multiple levels but the one that is easiest to see these things is DateRangePrefixTreeTest.java

          Show
          David Smiley added a comment - I'm glad you like it. 2000-00 is wrong because the displayed and toString January is 01, not 00. It's true that internally 0 is used. Arguably this should throw an exception but I suppose the current behavior is not unexpected. [2000 TO 2014] and [2000-01 to 2014-12] should be equivalent. FYI there are 3 tests at multiple levels but the one that is easiest to see these things is DateRangePrefixTreeTest.java
          Hide
          Varun Thacker added a comment -

          Thanks David for clearing it out my doubts.

          I added a patch which adds tests around 'dateRange' having multiple ranges.
          For this I made 'dateRange' field multiValued in the solrconfig.

          Show
          Varun Thacker added a comment - Thanks David for clearing it out my doubts. I added a patch which adds tests around 'dateRange' having multiple ranges. For this I made 'dateRange' field multiValued in the solrconfig.
          Hide
          ASF subversion and git services added a comment -

          Commit 1615961 from David Smiley in branch 'dev/trunk'
          [ https://svn.apache.org/r1615961 ]

          SOLR-6103: Test mult-valued DateRangeField (thanks Varun Thacker)

          Show
          ASF subversion and git services added a comment - Commit 1615961 from David Smiley in branch 'dev/trunk' [ https://svn.apache.org/r1615961 ] SOLR-6103 : Test mult-valued DateRangeField (thanks Varun Thacker)
          Hide
          David Smiley added a comment -

          Thanks Varun; committed now.
          FYI the tests at the Lucene level are very thorough and use the randomized testing mindset. So at the Solr level the Solr test may suggest it's not tested well but it's not the case at all.

          Show
          David Smiley added a comment - Thanks Varun; committed now. FYI the tests at the Lucene level are very thorough and use the randomized testing mindset. So at the Solr level the Solr test may suggest it's not tested well but it's not the case at all.
          Hide
          Varun Thacker added a comment -

          Indeed. Just went through DateNRStrategyTest.

          One use case which came to my mind - Specify recurring date ranges. Say summer vacations every year . Then one could boost certain type of documents in that period. Although I would wait till someone actually has a use case like this

          Show
          Varun Thacker added a comment - Indeed. Just went through DateNRStrategyTest. One use case which came to my mind - Specify recurring date ranges. Say summer vacations every year . Then one could boost certain type of documents in that period. Although I would wait till someone actually has a use case like this
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              David Smiley
              Reporter:
              David Smiley
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development