Solr
  1. Solr
  2. SOLR-3258

Ping query caused exception..Invalid version (expected 2, but 60) or the data in not in 'javabin' format

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Environment:

      solr-impl 4.0-SNAPSHOT 1302403 - markus - 2012-03-19 13:55:51

      Description

      In a test set-up with nodes=2, shards=3 and cores=6 we often see this exception in the logs. Once every few ping requests this is thrown, other request return a proper OK.

      Ping request handler:

      <requestHandler name="/admin/ping" class="solr.PingRequestHandler">
          <lst name="invariants">
            <str name="qt">select</str>
            <str name="q">*:*</str>
            <int name="rows">0</int>
          </lst>
          <lst name="defaults">
            <str name="wt">json</str>
            <str name="echoParams">all</str>
            <bool name="omitHeader">true</bool>
          </lst>
        </requestHandler>
      

      Exception:

      2012-03-20 13:16:06,405 INFO [solr.core.SolrCore] - [http-80-18] - : [core_a] webapp=/solr path=/admin/ping params={} status=500 QTime=7 
      2012-03-20 13:16:06,406 ERROR [solr.servlet.SolrDispatchFilter] - [http-80-18] - : null:org.apache.solr.common.SolrException: Ping query caused exception: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
              at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:77)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540)
              at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:435)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:256)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
              at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540)
              at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:68)
              ... 16 more
      Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
              at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:278)
              at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:158)
              at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:123)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              ... 1 more
      Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
              at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109)
              at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
              at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:468)
              at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
              at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:251)
              ... 10 more
      
      1. zkdump.txt
        266 kB
        Markus Jelsma
      2. debugging.patch
        2 kB
        Dawid Weiss

        Issue Links

          Activity

          Hide
          Dawid Weiss added a comment -

          I once tried to debug it but couldn't reproduce. It does happen from time to time on my build machine though.

          '60' is ASCII for '<' so I guess it's something weird emitted. Can you apply the attached patch and try to cause this, Markus?

          Show
          Dawid Weiss added a comment - I once tried to debug it but couldn't reproduce. It does happen from time to time on my build machine though. '60' is ASCII for '<' so I guess it's something weird emitted. Can you apply the attached patch and try to cause this, Markus?
          Hide
          Markus Jelsma added a comment -

          I've redeployed with your patch. This is very peculiar indeed! The stack trace shows a ping failing and at the bottom some that work well. I've also noticed the /select handler not being there so i've did a manual request on /select?q=: and i sometimes get the same error. Some work, some don't.

          Does this help a bit?

          2012-03-20 13:45:30,352 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=500 QTime=7 
          2012-03-20 13:45:30,352 ERROR [solr.servlet.SolrDispatchFilter] - [http-80-17] - : null:org.apache.solr.common.SolrException: Ping query caused exception: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>
                  at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:77)
                  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
                  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540)
                  at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:435)
                  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:256)
                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
                  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
                  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>
                  at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
                  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
                  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540)
                  at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:68)
                  ... 16 more
          Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>
                  at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:278)
                  at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:158)
                  at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:123)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  ... 1 more
          Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>
                  at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109)
                  at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
                  at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:468)
                  at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
                  at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:251)
                  ... 10 more
          
          2012-03-20 13:45:30,585 INFO [solr.core.SolrCore] - [http-80-3] - : [] webapp=/solr path=/select params={omitHeader=true&fl=id,score&shard.url=nl2.index.openindex.io:80/solr/|nl1.index.openindex.io:80/solr/openindex_a/&NOW=1332251130581&start=0&q=*:*&distrib=false&isShard=true&wt=javabin&fsv=true&rows=0&version=2} hits=0 status=0 QTime=0 
          2012-03-20 13:45:30,586 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=0 QTime=5 
          2012-03-20 13:45:30,587 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=0 QTime=6
          
          Show
          Markus Jelsma added a comment - I've redeployed with your patch. This is very peculiar indeed! The stack trace shows a ping failing and at the bottom some that work well. I've also noticed the /select handler not being there so i've did a manual request on /select?q= : and i sometimes get the same error. Some work, some don't. Does this help a bit? 2012-03-20 13:45:30,352 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=500 QTime=7 2012-03-20 13:45:30,352 ERROR [solr.servlet.SolrDispatchFilter] - [http-80-17] - : null :org.apache.solr.common.SolrException: Ping query caused exception: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size= "1" noshade= "noshade" ><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size= "1" noshade= "noshade" ><h3>Apache Tomcat/6.0.35</h3></body></html> at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:77) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:435) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:256) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang. Thread .run( Thread .java:662) Caused by: org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size= "1" noshade= "noshade" ><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size= "1" noshade= "noshade" ><h3>Apache Tomcat/6.0.35</h3></body></html> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540) at org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:68) ... 16 more Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size= "1" noshade= "noshade" ><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size= "1" noshade= "noshade" ><h3>Apache Tomcat/6.0.35</h3></body></html> at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:278) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:158) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:123) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ... 1 more Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format, input: <html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /solr/select</h1><HR size= "1" noshade= "noshade" ><p><b>type</b> Status report</p><p><b>message</b> <u>/solr/select</u></p><p><b>description</b> <u>The requested resource (/solr/select) is not available.</u></p><HR size= "1" noshade= "noshade" ><h3>Apache Tomcat/6.0.35</h3></body></html> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:468) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:251) ... 10 more 2012-03-20 13:45:30,585 INFO [solr.core.SolrCore] - [http-80-3] - : [] webapp=/solr path=/select params={omitHeader= true &fl=id,score&shard.url=nl2.index.openindex.io:80/solr/|nl1.index.openindex.io:80/solr/openindex_a/&NOW=1332251130581&start=0&q=*:*&distrib= false &isShard= true &wt=javabin&fsv= true &rows=0&version=2} hits=0 status=0 QTime=0 2012-03-20 13:45:30,586 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=0 QTime=5 2012-03-20 13:45:30,587 INFO [solr.core.SolrCore] - [http-80-17] - : [] webapp=/solr path=/admin/ping params={} status=0 QTime=6
          Hide
          Dawid Weiss added a comment -

          And here comes the moment where my knowledge of Solr ends I'd say there is definitely a bug in improper handling of HTTP response status (and this should be fixed), unless there is a filter somewhere that emits this HTML and fakes HTTP 200... But as for the cause of why this happens in general – no idea.

          Show
          Dawid Weiss added a comment - And here comes the moment where my knowledge of Solr ends I'd say there is definitely a bug in improper handling of HTTP response status (and this should be fixed), unless there is a filter somewhere that emits this HTML and fakes HTTP 200... But as for the cause of why this happens in general – no idea.
          Hide
          Markus Jelsma added a comment -

          Nasty!

          With this issue indexing fails, although some documents seem to be added. Custom request handlers still work but the default /select handler gives the trouble, which is used by our ping handler. Manual requests to /select?distrib=false do work without trouble.

          I also know that this happens with an empty index.

          I'd love to provide more details but i haven't. For now the issue is here but it just might disappear as suddenly as it appeared.

          Show
          Markus Jelsma added a comment - Nasty! With this issue indexing fails, although some documents seem to be added. Custom request handlers still work but the default /select handler gives the trouble, which is used by our ping handler. Manual requests to /select?distrib=false do work without trouble. I also know that this happens with an empty index. I'd love to provide more details but i haven't. For now the issue is here but it just might disappear as suddenly as it appeared.
          Hide
          Markus Jelsma added a comment -

          I suspected Solr's distributed capabilities because of the error occuring with distrib=true. So i stopped Zookeeper, removed the data directory and restarted Zookeeper and the Solr nodes. I attached a zookeeper dump i took just moments before removing the data directory.

          Show
          Markus Jelsma added a comment - I suspected Solr's distributed capabilities because of the error occuring with distrib=true. So i stopped Zookeeper, removed the data directory and restarted Zookeeper and the Solr nodes. I attached a zookeeper dump i took just moments before removing the data directory.
          Hide
          Yonik Seeley added a comment -

          I suspected Solr's distributed capabilities because of the error occuring with distrib=true.

          I was going to ask... do you mean for the ping query to be distributed?

          Show
          Yonik Seeley added a comment - I suspected Solr's distributed capabilities because of the error occuring with distrib=true. I was going to ask... do you mean for the ping query to be distributed?
          Hide
          Markus Jelsma added a comment -

          It seems it is. The ping query is just a /select?q=:&rows=0 but it yields different results with distrib=false specified.

          Show
          Markus Jelsma added a comment - It seems it is. The ping query is just a /select?q= : &rows=0 but it yields different results with distrib=false specified.
          Hide
          Markus Jelsma added a comment -

          It seems i found the problem. The solr.xml file on one of the nodes received a typo. Instead shard="shard1" i had shard_a="shard1". It's pretty hard to reproduce but after removing the ZK data directories you can start the nodes with one core having a bad shard parameter. Originally only one node had a corrupt solr.xml file but i could only reproduce by corrupting the file on both nodes and starting Solr.

          Show
          Markus Jelsma added a comment - It seems i found the problem. The solr.xml file on one of the nodes received a typo. Instead shard="shard1" i had shard_a="shard1". It's pretty hard to reproduce but after removing the ZK data directories you can start the nodes with one core having a bad shard parameter. Originally only one node had a corrupt solr.xml file but i could only reproduce by corrupting the file on both nodes and starting Solr.
          Hide
          Hoss Man added a comment -

          If i'm understanding correctly, it sounds like the root issue was invalid configuration on one server, and the error message was due to that javabin response codec trying to parse an plain text error response page from that server.

          So it doesn't seem like there is any bug to fix here?

          Show
          Hoss Man added a comment - If i'm understanding correctly, it sounds like the root issue was invalid configuration on one server, and the error message was due to that javabin response codec trying to parse an plain text error response page from that server. So it doesn't seem like there is any bug to fix here?
          Hide
          Dawid Weiss added a comment -

          I don't know if it's invalid configuration or something else but the error message could be less cryptic I mean – some kind of sanity check during header parsing would be nice (if possible).

          Show
          Dawid Weiss added a comment - I don't know if it's invalid configuration or something else but the error message could be less cryptic I mean – some kind of sanity check during header parsing would be nice (if possible).
          Hide
          Hoss Man added a comment -

          some kind of sanity check during header parsing would be nice (if possible).

          yeha ... but that's kind of orthoginal to the initial bug report (that ping wasn't working in distrib mode)

          I opened SOLR-3530 to look into improving things, but there may be diminishing returns here.

          Show
          Hoss Man added a comment - some kind of sanity check during header parsing would be nice (if possible). yeha ... but that's kind of orthoginal to the initial bug report (that ping wasn't working in distrib mode) I opened SOLR-3530 to look into improving things, but there may be diminishing returns here.

            People

            • Assignee:
              Unassigned
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development