Solr
  1. Solr
  2. SOLR-840

BinaryResponseWriter does not handle nulls with shards as it does locally

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Component/s: search
    • Labels:
      None
    • Environment:

      All

      Description

      When you query a number field
      locally, it can return null. However, when you go through a shard if
      you have an empty number it throws an error.

      I found wrapping the BinaryResponseWrite with a try/catch solved the
      problem and allows null values to be returned.

      BinaryResponseWriter.java:141
      try

      { val = useFieldObjects ? ft.toObject(f) : ft.toExternal(f); }

      catch (NumberFormatException e)

      { val = null; }

      It seems only the BinaryResponseWriter is actually that fussy about
      null items. Once it comes back to the client for display, it is
      handled without error.

      Hoss suggested this should be fixed for 1.3 but now that it has shipped I have marked it as 1.3.1 (
      See: http://www.nabble.com/best-way-to-debug-shard-format-errors-td19087854.html)

        Activity

        Hide
        Noble Paul added a comment -

        could you post the stacktrace as well?

        Show
        Noble Paul added a comment - could you post the stacktrace as well?
        Hide
        ian connor added a comment -

        HTTP ERROR: 500

        parsing error

        org.apache.solr.common.SolrException: parsing error
        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:385)
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183)
        at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:396)
        at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:369)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:675)
        Caused by: java.io.EOFException
        at org.apache.solr.common.util.FastInputStream.readByte(FastInputStream.java:160)
        at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:143)
        at org.apache.solr.common.util.NamedListCodec.readArray(NamedListCodec.java:367)
        at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:153)
        at org.apache.solr.common.util.NamedListCodec.readSolrDocumentList(NamedListCodec.java:301)
        at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:172)
        at org.apache.solr.common.util.NamedListCodec.readOrderedMap(NamedListCodec.java:98)
        at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:154)
        at org.apache.solr.common.util.NamedListCodec.unmarshal(NamedListCodec.java:89)
        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:39)
        ... 12 more

        RequestURI=/solr/select/

        Show
        ian connor added a comment - HTTP ERROR: 500 parsing error org.apache.solr.common.SolrException: parsing error at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:385) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:396) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:369) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:675) Caused by: java.io.EOFException at org.apache.solr.common.util.FastInputStream.readByte(FastInputStream.java:160) at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:143) at org.apache.solr.common.util.NamedListCodec.readArray(NamedListCodec.java:367) at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:153) at org.apache.solr.common.util.NamedListCodec.readSolrDocumentList(NamedListCodec.java:301) at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:172) at org.apache.solr.common.util.NamedListCodec.readOrderedMap(NamedListCodec.java:98) at org.apache.solr.common.util.NamedListCodec.readVal(NamedListCodec.java:154) at org.apache.solr.common.util.NamedListCodec.unmarshal(NamedListCodec.java:89) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:39) ... 12 more RequestURI=/solr/select/
        Hide
        Noble Paul added a comment -

        the problem apparently is with writing

        can you do

        try { val = useFieldObjects ? ft.toObject(f) : ft.toExternal(f); } catch (NumberFormatException e) {e.printStacktrace(); val = null; }
        

        and paste the stacktrace

        Show
        Noble Paul added a comment - the problem apparently is with writing can you do try { val = useFieldObjects ? ft.toObject(f) : ft.toExternal(f); } catch (NumberFormatException e) {e.printStacktrace(); val = null ; } and paste the stacktrace
        Hide
        ian connor added a comment -

        java.lang.NumberFormatException: For input string: ""
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:66)
        at java.lang.Integer.parseInt(Integer.java:497)
        at java.lang.Integer.valueOf(Integer.java:576)
        at org.apache.solr.schema.IntField.toObject(IntField.java:71)
        at org.apache.solr.schema.IntField.toObject(IntField.java:32)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.getDoc(BinaryResponseWriter.java:145)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:122)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:87)
        at org.apache.solr.common.util.NamedListCodec.writeVal(NamedListCodec.java:131)
        at org.apache.solr.common.util.NamedListCodec.writeNamedList(NamedListCodec.java:121)
        at org.apache.solr.common.util.NamedListCodec.marshal(NamedListCodec.java:80)
        at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:47)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
        at org.mortbay.jetty.Server.handle(Server.java:285)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
        java.lang.NumberFormatException: For input string: ""
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:66)
        at java.lang.Integer.parseInt(Integer.java:497)
        at java.lang.Integer.valueOf(Integer.java:576)
        at org.apache.solr.schema.IntField.toObject(IntField.java:71)
        at org.apache.solr.schema.IntField.toObject(IntField.java:32)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.getDoc(BinaryResponseWriter.java:145)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:122)
        at org.apache.solr.request.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:87)
        at org.apache.solr.common.util.NamedListCodec.writeVal(NamedListCodec.java:131)
        at org.apache.solr.common.util.NamedListCodec.writeNamedList(NamedListCodec.java:121)
        at org.apache.solr.common.util.NamedListCodec.marshal(NamedListCodec.java:80)
        at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:47)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
        at org.mortbay.jetty.Server.handle(Server.java:285)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

        Show
        ian connor added a comment - java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:66) at java.lang.Integer.parseInt(Integer.java:497) at java.lang.Integer.valueOf(Integer.java:576) at org.apache.solr.schema.IntField.toObject(IntField.java:71) at org.apache.solr.schema.IntField.toObject(IntField.java:32) at org.apache.solr.request.BinaryResponseWriter$Resolver.getDoc(BinaryResponseWriter.java:145) at org.apache.solr.request.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:122) at org.apache.solr.request.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:87) at org.apache.solr.common.util.NamedListCodec.writeVal(NamedListCodec.java:131) at org.apache.solr.common.util.NamedListCodec.writeNamedList(NamedListCodec.java:121) at org.apache.solr.common.util.NamedListCodec.marshal(NamedListCodec.java:80) at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:47) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:285) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:66) at java.lang.Integer.parseInt(Integer.java:497) at java.lang.Integer.valueOf(Integer.java:576) at org.apache.solr.schema.IntField.toObject(IntField.java:71) at org.apache.solr.schema.IntField.toObject(IntField.java:32) at org.apache.solr.request.BinaryResponseWriter$Resolver.getDoc(BinaryResponseWriter.java:145) at org.apache.solr.request.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:122) at org.apache.solr.request.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:87) at org.apache.solr.common.util.NamedListCodec.writeVal(NamedListCodec.java:131) at org.apache.solr.common.util.NamedListCodec.writeNamedList(NamedListCodec.java:121) at org.apache.solr.common.util.NamedListCodec.marshal(NamedListCodec.java:80) at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:47) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:285) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
        Hide
        Noble Paul added a comment -

        Thanks Ian
        So the problem is with the int field storing an empty string and it tries to convert that into an int. All other response writers write it down as String and never attempts to convert it to an int.

        I am wondering whether the fix must go into BinaryResponseWriter or IntField. For the time being the fix you have is good

        it can be bettered

        try { val = useFieldObjects ? ft.toObject(f) : ft.toExternal(f); } catch (Exception e) {continue; }
        

        Let us wait for the masters of Lucene to say what is the ideal fix?

        Show
        Noble Paul added a comment - Thanks Ian So the problem is with the int field storing an empty string and it tries to convert that into an int. All other response writers write it down as String and never attempts to convert it to an int. I am wondering whether the fix must go into BinaryResponseWriter or IntField. For the time being the fix you have is good it can be bettered try { val = useFieldObjects ? ft.toObject(f) : ft.toExternal(f); } catch (Exception e) { continue ; } Let us wait for the masters of Lucene to say what is the ideal fix?
        Hide
        Noble Paul added a comment -

        Yonik, Hoss can anybody explain how can an IntField contain an empty string as value? What should be the fix? in InfField or in BinaryResponseWriter?

        Show
        Noble Paul added a comment - Yonik, Hoss can anybody explain how can an IntField contain an empty string as value? What should be the fix? in InfField or in BinaryResponseWriter?
        Hide
        Noble Paul added a comment -

        catch and log the error

        Show
        Noble Paul added a comment - catch and log the error
        Hide
        Shalin Shekhar Mangar added a comment -

        Ian, is it possible that you changed the schema from a string to integer/sint and did not re-index? I'm not sure why an empty string will be there in an integer field. Anyway, we should handle this kind of error.

        Show
        Shalin Shekhar Mangar added a comment - Ian, is it possible that you changed the schema from a string to integer/sint and did not re-index? I'm not sure why an empty string will be there in an integer field. Anyway, we should handle this kind of error.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 718179.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 718179. Thanks Noble!
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            ian connor
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development