[LUCENE-2426] change sort order to binary order

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1
• Fix Version/s: 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 02/May/10 14:19
• Updated: 28/Aug/22 12:25
• Resolved: 24/Jun/10 13:36

Description

Since flexible indexing, terms are now represented as byte[], but for backwards compatibility reasons, they are not sorted as byte[], but instead as if they were char[].

I think its time to look at sorting terms as byte[]... this would yield the following improvements:

• terms are more opaque by default, they are byte[] and sort as byte[]. I think this would make lucene friendlier to customizations.
• numerics and collation are then free to use their own encoding (full byte) rather than avoiding the use of certain bits to remain compatible with char[] sort order.
• automaton gets simpler because as in LUCENE-2265, it uses byte[] too, and has special hacks because terms are sorted as char[]

Issue Links

• depends upon: LUCENE-2380 Add FieldCache.getTermBytes, to load term data as byte[]
• depends upon: LUCENE-2442 Remove flex back compat layers & pre-flex APIs
• depends upon: LUCENE-2265 improve automaton performance by running on byte[]
• is blocked by: LUCENE-2378 Cutover remaining usage of pre-flex APIs