Solr
  1. Solr
  2. SOLR-5936

Deprecate non-Trie-based numeric (and date) field types in 4.x and remove them from 5.0

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.8, 6.0
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      We've been discouraging people from using non-Trie numeric&date field types for years, it's time we made it official.

      1. SOLR-5936.branch_4x.patch
        12 kB
        Steve Rowe
      2. SOLR-5936.branch_4x.patch
        12 kB
        Steve Rowe
      3. SOLR-5936.branch_4x.patch
        9 kB
        Steve Rowe
      4. SOLR-5936.trunk.patch
        233 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Steve Rowe added a comment -

          4.x patch with deprecations.

          I'll work up a separate trunk patch.

          Show
          Steve Rowe added a comment - 4.x patch with deprecations. I'll work up a separate trunk patch.
          Hide
          Steve Rowe added a comment -

          Here's an updated 4.x patch that removes these field types from the schemaless example (users almost certainly won't have existing indexes with the legacy numeric/date types), and adds comments to the main example schema about these types being deprecated and removed in 5.0.

          The DIH example schemas are waaaay out of date - they don't include any of the trie fields - I'll make a separate issue to clean them up.

          Show
          Steve Rowe added a comment - Here's an updated 4.x patch that removes these field types from the schemaless example (users almost certainly won't have existing indexes with the legacy numeric/date types), and adds comments to the main example schema about these types being deprecated and removed in 5.0. The DIH example schemas are waaaay out of date - they don't include any of the trie fields - I'll make a separate issue to clean them up.
          Hide
          Jack Krupansky added a comment -

          As part of this cleanup, could somebody volunteer to create a plain-English summary of exactly what a trie field really is, what good it is, and why we can't live without them? I've read the code and, okay, there is a sequence of bit shifts and generation of extra terms, but in plain English, what's the point?

          I'm not asking for a recitation of the actual algorithm(s), but some intuitively accessible summary. I would note that the typical examples are for strings with prefixes rather than binary numbers.

          See:
          http://en.wikipedia.org/wiki/Trie

          And, is trie really the best solution for number types? Does it actually have real value for float and double values?

          And I would really like to see some plain, easily readable explanation of precision step. Again, especially for real numbers.

          And how should precision step be used for dates?

          I mean, other than assuring sort order, why bother with trie? Or more specifically, why does a Solr (or Lucene) user need to know that trie is used for the implementation?

          Specifically, for example, does it matter if a field has an evenly distributed range of numeric values with little repetition vs. numeric codes where there is a relatively small number of distinct values (e.g., 1-10, or scores of 0-100 or dates in years between 1970 and 2014) and relatively high cardinality? I mean, does trie do a uniformly great job for both of these extreme use cases, including for faceting?

          And if trie really is the best approach for numeric fields, why not just do all of this under the hood instead of polluting the field type names with "trie"? IOW, rename TrieIntField to IntField, etc.

          To me, trie just seems like unnecessary noise to average users.

          Show
          Jack Krupansky added a comment - As part of this cleanup, could somebody volunteer to create a plain-English summary of exactly what a trie field really is, what good it is, and why we can't live without them? I've read the code and, okay, there is a sequence of bit shifts and generation of extra terms, but in plain English, what's the point? I'm not asking for a recitation of the actual algorithm(s), but some intuitively accessible summary. I would note that the typical examples are for strings with prefixes rather than binary numbers. See: http://en.wikipedia.org/wiki/Trie And, is trie really the best solution for number types? Does it actually have real value for float and double values? And I would really like to see some plain, easily readable explanation of precision step. Again, especially for real numbers. And how should precision step be used for dates? I mean, other than assuring sort order, why bother with trie? Or more specifically, why does a Solr (or Lucene) user need to know that trie is used for the implementation? Specifically, for example, does it matter if a field has an evenly distributed range of numeric values with little repetition vs. numeric codes where there is a relatively small number of distinct values (e.g., 1-10, or scores of 0-100 or dates in years between 1970 and 2014) and relatively high cardinality? I mean, does trie do a uniformly great job for both of these extreme use cases, including for faceting? And if trie really is the best approach for numeric fields, why not just do all of this under the hood instead of polluting the field type names with "trie"? IOW, rename TrieIntField to IntField, etc. To me, trie just seems like unnecessary noise to average users.
          Hide
          Uwe Schindler added a comment - - edited

          Hi Jack,

          And if trie really is the best approach for numeric fields, why not just do all of this under the hood instead of polluting the field type names with "trie"? IOW, rename TrieIntField to IntField, etc.

          This goes back to the introduction of that in Lucene 2.9 / Solr 1.4. At that time everybody was using other field types, and stuff like IntField, SortableIntField,.. was already used as names. Because of that it was introduced to Solr with the name based on the original donated code (by me). Shortly later, Lucene renamed the field to be "NumericField" and "NumericRangeQuery" the query. The term "trie" is no longer used in Lucene and only the term "precisionStep" as a configureable flag for the number of additional term remained (in the documentation). So "Trie(Int|Long|Float|Double|Date)Field" is just there for "backwards compatibility" with earlier indexes (in Solr 1.4) and now, because the name is baked in, no way to change anymore.

          +1 to rename for 5.0

          As part of this cleanup, could somebody volunteer to create a plain-English summary of exactly what a trie field really is, what good it is, and why we can't live without them? I've read the code and, okay, there is a sequence of bit shifts and generation of extra terms, but in plain English, what's the point?

          See javadocs of NumericRangeQuery.

          Specifically, for example, does it matter if a field has an evenly distributed range of numeric values with little repetition vs. numeric codes where there is a relatively small number of distinct values (e.g., 1-10, or scores of 0-100 or dates in years between 1970 and 2014) and relatively high cardinality?

          This does not matter because of the structure of the additional terms. The number of terms used for actual ranges is almost always around the approx. expected number (see javadocs of NRQ). It also does not matter if it is a date or a int or a float. Internally, for trie, there are no floats or dates at all. Everything is mapped to the sortable bits (means if value_a < value_b also the bits_of_value_a < bits_of_value_b). It also has no real effect on the size of the range. Lucene always matches approximately the same number of terms (a few hundreds at maximum).

          Simply said, you are indexing all numbers as bits like strings formed as "10110110" (just in a better compressed way), with additional terms stripping some bits from the right (like "10110110", "101101", "1011", "10"). Ranges are then simplified to match middle parts of the range with shorter terms that match more documents. For that algorithm, the distribution of values is not that important. Index size only grows by a minimum size, because the shorter terms are more rare (approx. 12% more terms), with large posting lists (many docs match). But as those terms match many sequential docs, the posting lists are not so big (because of the delta encoding). So trie terms raise the index size only by a few percents, but make range queries ultimatively fast, because ranges can be matched with few terms hitting many documents.

          See: http://en.wikipedia.org/wiki/Trie

          This wikipedia page does not apply to numeric fields in Lucene. The term "Trie" only comes from the structure of the terms generated by the NumericTokenStream. But the algorithm is not the same as for trie lookups. It is just a name, nothing meaningful about the implementation. So I agree, we should nuke it from the source (only Solr is using it, Lucene and Elasticsearch don't use it). I just wanted to have a name for "my algorithm and data structure mix" at that time.

          I mean, does trie do a uniformly great job for both of these extreme use cases, including for faceting?

          It is not used for facetting. Facetting does not use the additional terms. For facetting use DocValues instead of indexed fields. If you want to use Trie fields, and don't want to search on them with ranges, you can switch of the additional terms by setting precStep to 0.

          One last note from my side:
          I agree with removing the impl details from the user. The user in my opinion only needs 2 types of numerics: precisionStep=4 or 8 (I think the default in solr is 8, although I disagree - e.g., Elasticsearch uses the Lucene default of 4) and another one with precisonStep=infinity (0 in solr would) for numerics that are only for sorting and don't need range queries.

          Show
          Uwe Schindler added a comment - - edited Hi Jack, And if trie really is the best approach for numeric fields, why not just do all of this under the hood instead of polluting the field type names with "trie"? IOW, rename TrieIntField to IntField, etc. This goes back to the introduction of that in Lucene 2.9 / Solr 1.4. At that time everybody was using other field types, and stuff like IntField, SortableIntField,.. was already used as names . Because of that it was introduced to Solr with the name based on the original donated code (by me). Shortly later, Lucene renamed the field to be "NumericField" and "NumericRangeQuery" the query. The term "trie" is no longer used in Lucene and only the term "precisionStep" as a configureable flag for the number of additional term remained (in the documentation). So "Trie(Int|Long|Float|Double|Date)Field" is just there for "backwards compatibility" with earlier indexes (in Solr 1.4) and now, because the name is baked in, no way to change anymore. +1 to rename for 5.0 As part of this cleanup, could somebody volunteer to create a plain-English summary of exactly what a trie field really is, what good it is, and why we can't live without them? I've read the code and, okay, there is a sequence of bit shifts and generation of extra terms, but in plain English, what's the point? See javadocs of NumericRangeQuery. Specifically, for example, does it matter if a field has an evenly distributed range of numeric values with little repetition vs. numeric codes where there is a relatively small number of distinct values (e.g., 1-10, or scores of 0-100 or dates in years between 1970 and 2014) and relatively high cardinality? This does not matter because of the structure of the additional terms. The number of terms used for actual ranges is almost always around the approx. expected number (see javadocs of NRQ). It also does not matter if it is a date or a int or a float. Internally, for trie, there are no floats or dates at all. Everything is mapped to the sortable bits (means if value_a < value_b also the bits_of_value_a < bits_of_value_b). It also has no real effect on the size of the range. Lucene always matches approximately the same number of terms (a few hundreds at maximum). Simply said, you are indexing all numbers as bits like strings formed as "10110110" (just in a better compressed way), with additional terms stripping some bits from the right (like "10110110", "101101", "1011", "10"). Ranges are then simplified to match middle parts of the range with shorter terms that match more documents. For that algorithm, the distribution of values is not that important. Index size only grows by a minimum size, because the shorter terms are more rare (approx. 12% more terms), with large posting lists (many docs match). But as those terms match many sequential docs, the posting lists are not so big (because of the delta encoding). So trie terms raise the index size only by a few percents, but make range queries ultimatively fast, because ranges can be matched with few terms hitting many documents. See: http://en.wikipedia.org/wiki/Trie This wikipedia page does not apply to numeric fields in Lucene. The term "Trie" only comes from the structure of the terms generated by the NumericTokenStream. But the algorithm is not the same as for trie lookups. It is just a name, nothing meaningful about the implementation. So I agree, we should nuke it from the source (only Solr is using it, Lucene and Elasticsearch don't use it). I just wanted to have a name for "my algorithm and data structure mix" at that time. I mean, does trie do a uniformly great job for both of these extreme use cases, including for faceting? It is not used for facetting. Facetting does not use the additional terms. For facetting use DocValues instead of indexed fields. If you want to use Trie fields, and don't want to search on them with ranges, you can switch of the additional terms by setting precStep to 0. One last note from my side: I agree with removing the impl details from the user. The user in my opinion only needs 2 types of numerics: precisionStep=4 or 8 (I think the default in solr is 8, although I disagree - e.g., Elasticsearch uses the Lucene default of 4) and another one with precisonStep=infinity (0 in solr would) for numerics that are only for sorting and don't need range queries.
          Hide
          Erick Erickson added a comment -

          Could we take the pint, plong, pfloat and all that out of the example schema while we're at it? Maybe in trunk only? I think that trunk, at least, won't have to read indexes with these it it.

          Show
          Erick Erickson added a comment - Could we take the pint, plong, pfloat and all that out of the example schema while we're at it? Maybe in trunk only? I think that trunk, at least, won't have to read indexes with these it it.
          Hide
          Steve Rowe added a comment -

          Could we take the pint, plong, pfloat and all that out of the example schema while we're at it? Maybe in trunk only? I think that trunk, at least, won't have to read indexes with these it it.

          +1 - see the issue title

          Show
          Steve Rowe added a comment - Could we take the pint, plong, pfloat and all that out of the example schema while we're at it? Maybe in trunk only? I think that trunk, at least, won't have to read indexes with these it it. +1 - see the issue title
          Hide
          Yonik Seeley added a comment -

          I think we can remove this from the example schema alltogether?

          +   <!-- The pint (solr.IntField) field type is deprecated and will be removed in Solr 5.0! -->
              <dynamicField name="*_pi"  type="pint"    indexed="true"  stored="true"/>
          

          I had added it a long time ago for manual testing purposes.

          Show
          Yonik Seeley added a comment - I think we can remove this from the example schema alltogether? + <!-- The pint (solr.IntField) field type is deprecated and will be removed in Solr 5.0! --> <dynamicField name= "*_pi" type= "pint" indexed= " true " stored= " true " /> I had added it a long time ago for manual testing purposes.
          Hide
          Steve Rowe added a comment -

          I think we can remove this from the example schema alltogether?

          +   <!-- The pint (solr.IntField) field type is deprecated and will be removed in Solr 5.0! -->
              <dynamicField name="*_pi"  type="pint"    indexed="true"  stored="true"/>
          

          I had added it a long time ago for manual testing purposes.

          This patch removes it.

          Show
          Steve Rowe added a comment - I think we can remove this from the example schema alltogether? + <!-- The pint (solr.IntField) field type is deprecated and will be removed in Solr 5.0! --> <dynamicField name= "*_pi" type= "pint" indexed= " true " stored= " true " /> I had added it a long time ago for manual testing purposes. This patch removes it.
          Hide
          Yonik Seeley added a comment -

          +1 to commit last patch

          Show
          Yonik Seeley added a comment - +1 to commit last patch
          Hide
          Steve Rowe added a comment -

          I'm going to commit the 4.x patch now - the SOLR-5937 changes will depend on the main example schema changes here.

          Show
          Steve Rowe added a comment - I'm going to commit the 4.x patch now - the SOLR-5937 changes will depend on the main example schema changes here.
          Hide
          ASF subversion and git services added a comment -

          Commit 1583179 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1583179 ]

          SOLR-5936: Deprecate non-Trie-based numeric & date field types

          Show
          ASF subversion and git services added a comment - Commit 1583179 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1583179 ] SOLR-5936 : Deprecate non-Trie-based numeric & date field types
          Hide
          ASF subversion and git services added a comment -

          Commit 1583226 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1583226 ]

          SOLR-5936: don't use deprecated solr.IntField field type in Solr example tests (dynamic field '*_pi', using solr.IntField via the 'pint' field type, was removed from the main example schema)

          Show
          ASF subversion and git services added a comment - Commit 1583226 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1583226 ] SOLR-5936 : don't use deprecated solr.IntField field type in Solr example tests (dynamic field '*_pi', using solr.IntField via the 'pint' field type, was removed from the main example schema)
          Hide
          Hoss Man added a comment -

          +1 to rename for 5.0

          What exactly do you suggest renaming these Solr FieldType's to?

          If you are suggesting "TrieFooField -> FooField" then i am a HUGE -1 to that idea.

          It's one thing to say that things like the (text based) IntField is deprecated, and will not work in 5.0 and people have to reindex. but if we also rename TrieIntField to IntField, then people who are still using the (text based) IntField in their schema.xml and attempt upgrading will get really weird, and hard to understand errors.

          If folks think Trie is a confusing word in the name and want to change that then fine – I'm certainly open to the idea – But we really should not re-use the name of an existing (deprecated/removed) field type in a way that isn't backcompat.


          In any event, a lot of what's being discussed here in comments feels like it should really be tracked in discreet issues (these can all be dealt with independnet of this issue, and eachother):

          • better jdocs for the trie numeric fields
          • renaming the trie numeric fields
          • simplifying configuration of the trie numeric fields

          ...let's please keep this issue focused on the deprecation & removal of the non-trie fields, and folks who care about these other idea can file other jira's to track them

          Show
          Hoss Man added a comment - +1 to rename for 5.0 What exactly do you suggest renaming these Solr FieldType's to? If you are suggesting "TrieFooField -> FooField" then i am a HUGE -1 to that idea. It's one thing to say that things like the (text based) IntField is deprecated, and will not work in 5.0 and people have to reindex. but if we also rename TrieIntField to IntField, then people who are still using the (text based) IntField in their schema.xml and attempt upgrading will get really weird, and hard to understand errors. If folks think Trie is a confusing word in the name and want to change that then fine – I'm certainly open to the idea – But we really should not re-use the name of an existing (deprecated/removed) field type in a way that isn't backcompat. In any event, a lot of what's being discussed here in comments feels like it should really be tracked in discreet issues (these can all be dealt with independnet of this issue, and eachother): better jdocs for the trie numeric fields renaming the trie numeric fields simplifying configuration of the trie numeric fields ...let's please keep this issue focused on the deprecation & removal of the non-trie fields, and folks who care about these other idea can file other jira's to track them
          Hide
          Steve Rowe added a comment - - edited

          Patch removing non-Trie numeric/date field types from trunk.

          Note that the example-DIH schemas are not addressed here - I'll do that under SOLR-5937 once I've committed here.

          Run the following bash script before applying the patch:

          for a in BCD{Int,Long,Str}Field.java DateField.java {Sortable,}{Double,Long,Float,Int}Field.java ; do
               svn rm solr/core/src/java/org/apache/solr/schema/$a
          done
          svn rm solr/core/src/java/org/apache/solr/util/BCDUtils.java
          svn rm solr/core/src/test/org/apache/solr/util/TestNumberUtils.java
          

          Solr tests pass, after several fun-filled days of legacy-field-type whack-a-mole...

          I think it's ready. I'll commit tomorrow unless there are objections.

          Show
          Steve Rowe added a comment - - edited Patch removing non-Trie numeric/date field types from trunk. Note that the example-DIH schemas are not addressed here - I'll do that under SOLR-5937 once I've committed here. Run the following bash script before applying the patch: for a in BCD{Int,Long,Str}Field.java DateField.java {Sortable,}{Double,Long,Float,Int}Field.java ; do svn rm solr/core/src/java/org/apache/solr/schema/$a done svn rm solr/core/src/java/org/apache/solr/util/BCDUtils.java svn rm solr/core/src/test/org/apache/solr/util/TestNumberUtils.java Solr tests pass, after several fun-filled days of legacy-field-type whack-a-mole... I think it's ready. I'll commit tomorrow unless there are objections.
          Hide
          ASF subversion and git services added a comment -

          Commit 1586106 from sarowe@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1586106 ]

          SOLR-5936: Removed deprecated non-Trie-based numeric & date field types.

          Show
          ASF subversion and git services added a comment - Commit 1586106 from sarowe@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1586106 ] SOLR-5936 : Removed deprecated non-Trie-based numeric & date field types.
          Hide
          Steve Rowe added a comment -

          Committed the deprecation patch on branch_4x; committed the removal patch on trunk; and committed the branch_4x and trunk versions of the example-DIH configset modernization sub-task (SOLR-5937).

          Show
          Steve Rowe added a comment - Committed the deprecation patch on branch_4x; committed the removal patch on trunk; and committed the branch_4x and trunk versions of the example-DIH configset modernization sub-task ( SOLR-5937 ).
          Hide
          Uwe Schindler added a comment -

          Close issue after release of 4.8.0

          Show
          Uwe Schindler added a comment - Close issue after release of 4.8.0

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Steve Rowe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development