Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13285

ByteArrayUtf8CharSequence cannot be cast to java.lang.String exception during replication

    XMLWordPrintableJSON

    Details

      Description

      Since upgrading to 7.7 (also tried 7.7.1, and 8.1.0) from 6.6.4, we're seeing the following errors in the SolrCloud elected master for a given collection when updates are written. This was after a full reindex of data (fresh build).

      request: http://solr-1.search-solr.preprod.k8.atcloud.io:80/solr/at-uk_shard1_replica_n2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsolr-2.search-solr.preprod.k8.atcloud.io%3A80%2Fsolr%2Fat-uk_shard1_replica_n1%2F&wt=javabin&version=2
      Remote error message: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:385) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:183) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
          at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
      

      Following this through to the replica, you'll see:

      08:35:22.060 [qtp1540374340-20] ERROR org.apache.solr.servlet.HttpSolrCall - null:java.lang.ClassCastException: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.common.util.JavaBinCodec.readEnumFieldValue(JavaBinCodec.java:813)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:339)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:640)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:337)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readMapEntry(JavaBinCodec.java:819)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:341)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:295)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
          at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
          at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
          at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
          at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
          at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.Server.handle(Server.java:502)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
          at java.lang.Thread.run(Thread.java:748)
      

      I noticed there some similar issues raised in 7x (https://issues.apache.org/jira/browse/SOLR-13255), so I don't know if this is the same sort of thing?

      I've tried solr-users mailing list, and stackoverflow but to no avail. Please let me know if there is any other information I can provide.

        Attachments

        1. SOLR-13285.patch
          3 kB
          Noble Paul
        2. SOLR-13285.patch
          3 kB
          Karl Stoney

          Issue Links

            Activity

              People

              • Assignee:
                noble.paul Noble Paul
                Reporter:
                kstoney Karl Stoney
              • Votes:
                3 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: