solr can't perform sort by long field

• Key: SOLR-324
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Minor
• Assignee: Grant Ingersoll
• Reporter: Sergey Dryganets
• Component/s: search
• Fix Version/s: 1.3

Environment

Windows XP SP2
Java 1.6u1
solr - lastes svn version

Description

when sorting results by field with type long
throwed folowing exception

java.lang.NumberFormatException: For input string: "1104836394000"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at org.apache.lucene.search.FieldCacheImpl$1.parseInt(FieldCacheImpl.java:136)
at org.apache.lucene.search.FieldCacheImpl$3.createValue(FieldCacheImpl.java:171)
at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:154)
at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:148)
at org.apache.lucene.search.FieldSortedHitQueue.comparatorInt(FieldSortedHitQueue.java:204)
at org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:175)
at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
at org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:155)
at org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:857)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:805)
at org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.java:698)
at org.apache.solr.handler.StandardRequestHandler.handleRequestBody(StandardRequestHandler.java:126)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:78)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:722)
at org.apache.solr.servlet.SolrServlet.doGet(SolrServlet.java:64)
at org.apache.solr.servlet.SolrServlet.doPost(SolrServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:187)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Unknown Source)

Comments

[Yonik Seeley]

This stems from lucene not being able to sort on a long field.
As a workaround for now, the type "slong" (sortable long), is able to do both range queries and sorts.

[Hoss Man]

Either LongField and DoubleField should log a warning that they not suitable for sorting (and this should be added to the javadocs) and that they defer to int/float based sorting which may have casting/parsing issues ... in which case people using small enough values will be fine.

...OR...

LongField and DoubleField should throw big exceptions if you try to use them to sort saying that if you wnat to Sort, use SortableLongField/SortableDoubleField

(the names seem pretty intuitive to me)

[Grant Ingersoll]

It also can't perform FunctionQuery's on longs and doubles either

[Grant Ingersoll]

Adding the functionality to the FieldCache should allow us to then implement the appropriate pieces in Solr: LongFieldSource, etc.

[Grant Ingersoll]

Requires LUCENE-1015 to be built and the respective Lucene jars to be brought into the lib directory and the old Lucene jars to be removed. This is a first draft, I still need to figure out how best to test it.

[Grant Ingersoll]

Patch based on latest trunk of Lucene, where I committed the changes to create ExtendedFieldCache and added support for sorting by long and doubles.

Also needs the latest Lucene jars.

[Grant Ingersoll]

There may be some changes to FieldCache based on this, which could remove the ExtendedFieldCache interface.

[Grant Ingersoll]

Update of this for the latest changes in Lucene (see LUCENE-1045)

Adds in support for Short and byte as well, but I did not add the declaration of those field types to the schema.xml examples.

Requires the latest trunk of Lucene (not even last nights build)

[Stu Hood]

Using long values for dates would be a significant improvement over the current String implementation.

[Otis Gospodnetic]

Grant, should this issue be closed? See LUCENE-1045

[Grant Ingersoll]

No, Solr needs to be able to leverage what is in LUCENE-1045. Thinks like a LongField need to be changed. The Patch should handle it, but it may need to be updated.

[Otis Gospodnetic]

Ah, this is in Solr-land, sorry, thought it was Lucene.

[Grant Ingersoll]

Here's an update to this patch. I dropped short/byte support for now because I somehow forgot to add in SortField.BYTE and SortField.SHORT on the related Lucene patch. I will open a separate issue for those.

Applying this patch should resolve the long/double issue.

[Grant Ingersoll]

Committed revision 629334.