OK let's open a new issue for how to best integrate/default SortField
Nevertheless, I would like to remove emphasis from NumericUtils (which is in realyity a helper class).
For bytes, TrieRange is not very interesting, for shorts, maybe, but I would subsume them during indexing as simple integers. You could not speedup searching, but limit index size a little bit.
Well, a RangeQuery on a "plain text" byte or short field requires
sneakiness (knowing that you must zero-pad; keeping
document.NumberUtils around); I think it's best if NumericXXX in
Lucene handles all of java's native numeric types. And you want a
byte or short out of FieldCache (to not waste RAM having to
upgrade to an int).
We can do this under the (a?) new issue too...
The SortField factory is then the only parts really needed in NumericUtils, but not really. The parser is a singleton, works for all trie fields and could also live somewhere else or nowhere at all, if the Parsers all stay in FieldCache.
(Under a new issue, but...) I'm not really a fan of leaving the parser
in FieldCache and expecting a user to "know" to create the SortField
with that parser. NumericSortField would make it much more consumable
to "direct" Lucene users.
Can we rename RangeQuery -> TextRangeQuery (TermRangeQuery), to make it clear that its range checking is by Term sort order.
We can do this and deprecate the old one, but I added a note to Javadocs (see patch). I would do this outside of this issue.
One benefit of a rename is it's a reminder to users on upgrading to
consider whether they should in fact switch to NumericRangeQuery.
How about oal.util.NumericUtils instead of TrieUtils?
That was my first idea, too. What to do with o.a.l.doc.NumberTools (deprecate?). And also update contrib/spatial to use NumericUtils instead of the copied and not really goo NumberUtils from Solr (Yonik said, it was written at a very early stage, and is not effective with UTF-8 encoding and the TermEnum posioning with the term prefixes). It would be a index-format change for spatial, but as the code was not yet released (in Lucene), the Lucene version should not use NumberUtils at all.
+1 on both (if we can add byte/short to trie*); we should do this
before 2.9 since we can still change locallucene's format. Maybe open
a new issue for that, too? We're forking off new 2.9 issues left and
I think, I remove the ShiftAttribute in complete, its really useless. Maybe, I add a getShift() method to NumericUtils, that returns the shift value of a Token/String. See java-dev mailing with Yonik.
Did you think about / decide against making a NumericField (that'd set the right tokenStream itself)?
Field is final and so I must extend AbstractField. But some methods of Document return Field and not AbstractField.
Can we just un-final Field?
NumericField would only work for indexing, but when retrieving from index (stored fields), it would change to Field.
Maybe we should move this after the index-specific schemas and so on. Or document, that it can be only used for indexing.
True, but we already have such "challenges" between index vs search
time Document; documenting it it seems fine.
By the way: How do you like the factories in NumericRangeQuery and the setValue methods, working like StringBuffer.append() in NumericTokenStream? This makes it really easy to index.
I think this is great! I like that you return NumericTokenStream
The only good thing of NumericField would be the possibility to automatically disable TF and Norms per default when indexing.
Consumability (good defaults)! (And also not having to know that you
must go and get a tokenStream from NumericUtils).