Sorting does not work correcly on "String Date"

• Key: LUCENE-463
• Type: Bug
• Status: Closed
• Resolution: Invalid
• Priority: Minor
• Assignee: Yonik Seeley
• Reporter: Etienne

Description

I am doing a simple query with a sort on a date field already put on a string format (YYYYmmDDHHMMSS). I am presenting 10 results per pages. The results I have is not correctly sort. On a page "4 october" is before "5 october", on another page I got the same kind of error.

These are logs of the sorting I get:

20051006102501 DATE: 06 oct. 2005
20051004130501 DATE: 04 oct. 2005 <-- PROBLEM!
20051005120200 DATE: 05 oct. 2005
20050928094805 DATE: 28 sept. 2005
20050928114901 DATE: 28 sept. 2005
20050928114901 DATE: 28 sept. 2005
20050928101901 DATE: 28 sept. 2005
20050928142601 DATE: 28 sept. 2005
20050928142601 DATE: 28 sept. 2005
20050929094100 DATE: 29 sept. 2005 <-- PROBLEM!

20050928132931 DATE: 28 sept. 2005
20050929090002 DATE: 29 sept. 2005 <-- PROBLEM!
20050927203403 DATE: 27 sept. 2005
20050927203402 DATE: 27 sept. 2005
20050927131901 DATE: 27 sept. 2005
20050927131901 DATE: 27 sept. 2005
20050927132501 DATE: 27 sept. 2005
20050927132500 DATE: 27 sept. 2005
20050927133101 DATE: 27 sept. 2005
20050927133100 DATE: 27 sept. 2005

20050926115001 DATE: 26 sept. 2005
20050927112301 DATE: 27 sept. 2005
20050926091200 DATE: 26 sept. 2005 <-- PROBLEM!
20050927085701 DATE: 27 sept. 2005
20050923111558 DATE: 23 sept. 2005
20050923103700 DATE: 23 sept. 2005
20050923144300 DATE: 23 sept. 2005
20050922102200 DATE: 22 sept. 2005
20050922082701 DATE: 22 sept. 2005
20050923085601 DATE: 23 sept. 2005 <-- PROBLEM!

20050815102204 DATE: 15 august 2005
20050815102201 DATE: 15 august 2005
20050816102200 DATE: 16 august 2005 <-- PROBLEM!
20050815110400 DATE: 15 august 2005
20050816124500 DATE: 16 august 2005 <-- PROBLEM!
20050812163334 DATE: 12 august 2005
20050812163331 DATE: 12 august 2005
20050812161840 DATE: 12 august 2005
20050812161935 DATE: 12 août 2005
20050812113442 DATE: 12 août 2005

Comments

[Hoss Man]

without providing any any explanation of how you are building the index, or how exactly you are executing your searches, I'm not sure that there is any way to understand the problem you are seeing

Please attach some code that demonstrates the probem.

Ideally this should be in the form of a JUnit Test, so it can easily be incorperated into the existing unit tests – but at a minimum people will need to see code for a standalone program with a main function that requires no input and builds up an index in either a RAMDirectory or in the current working directory and does a search/sort using the same method you are using in order to figure out what the problem might be.

[Cheolgoo Kang]

I've resolved the same problem today, and think it's the problem caused by Integer.parseInt(). For example, at the second line, Integer.parseInt( "20051004130501" ) gives an NumberFormatException with contained message "For input string: "20051004130501"".

So, I've made some changes of using Long.parseLong() rather than Integer.parseInt() to the classes related like FieldCache.java, FieldCacheImpl.java, and FieldSortedHitQueue.java.

And now it sorts date values in yyyymmddhhmmsszzz like a charm!

I can make some patches to address this issue right away, if there is anyone who needs it.

And of course, the performance and memory issue should be tested thoroughly.

And also, does anybody have a problem with sorting values in double, not float?

[Yonik Seeley]

Why not just use a string sort rather than an integer (or long) sort?

[Cheolgoo Kang]

AFAIK, sorting as number is far more faster than string. And friendly with memory issue also.

Some developers want to use the value they've inserted in database, and thinks it's wasting disk space to save another 'same' field just for sorting to the database or lucene index while they can sort it with existing values.

[Etienne]

Thanks for resolving it. I didn't get your first note asking me to send you the Junit test. Sorry, if I didn't respond. I still can build the unit test, for my use, and send it here. Could you make a patch (a lucene.jar)? I will work on it this week, and test it.

Thanks.

Etienne.

[Yonik Seeley]

Etienne, how are you specifying the sort? A string sort should work for this.

Cheolgoo, string sorting with no locale is faster than numeric sorting the first time you sort on a particular field. After the first time, string sorting will be the same speed as integer sorting, but faster than float sorting. The reason is that the strings are already sorted in the lucene index, so their index-order (ordinals) are compared. The string values themselves are never compared unless you are using MultiSearcher or if you specify a different Locale to sort by.

String sorting does use more memory, since an additional array of the unique terms is kept. See FieldCache.StringIndex. Unless a significant amount of documents have unique values, a long[] in the fieldcache could take more room.

[Etienne]

Hi,

the way I do the sort is something like this :

String sortingField = "date";
boolean sorder = (vo.getSorder().equals(LuceneConstants.ASCENDING))?true:false;
sorting = new Sort(sortingField, sorder); // where sorting field == "dateToBeSort"
Hits h = s.search(finalQuery, sorting);

And the field is stored like this in the Index:

public static void addFieldDate(Document luceneDoc, Date date, String field){
if(date!=null){ SimpleDateFormat formatter = new SimpleDateFormat(MetaDataConstants.DATE_FORMAT); String dateStr = formatter.format(date); luceneDoc.add(Field.Keyword(field,dateStr)); // where field is "dateToBeSort" }
}

So it seems it is already a String that I stored?

Etienne.

[Yonik Seeley]

> sorting = new Sort(sortingField, sorder);

Ahhh, that could be the problem. You aren't specifying what type to treat the field as, so it defaults to AUTO, meaning that lucene tries to figure out the field type at sort time by the first term for that field in the index.

Use the Sort constructor that takes a SortField, and specify string sorting.

[Yonik Seeley]

Yep, I verified that AUTO sorting is your problem.
In FieldCacheImpl for an AUTO type, first Integer.parseInt() is tried, and if that fails,
Float.parseFloat() is tried.
For your case, parseInt() will fail because it's too big, but parseFloat() will work (but not with enough precision).

Float.parseFloat("20051004130501")=2.00510045E13
Float.parseFloat("20051006102501")=2.00510066E13
Float.parseFloat("20051005102501")=2.00510045E13

Notice how two of the strings map to the same float when parsed as such.

[Yonik Seeley]

Closing. You can reopen if string sorting doesn't solve the problem for you.

[Etienne]

Hi,
Thanks all,

I think everything works well now. This way to make a sort is not mention in the Lucene In Action book. Maybe it should be a bit more detailled there.

public Sort createSorting(ILuceneFilterParameterVO vo){
String fieldToSort = vo.getSort();
Sort sorting = null;
if(fieldToSort!=null&&!fieldToSort.equals(LuceneConstants.NOSORT)){ // descending is the default setting for our requierements. boolean reverse = (vo.getSorder().equals(LuceneConstants.ASCENDING))?false:true; SortField sortField = new SortField(fieldToSort, SortField.STRING, reverse); sorting = new Sort(sortField); }
return sorting;
}

Thanks all.

Etienne.