[SOLR-13285] ByteArrayUtf8CharSequence cannot be cast to java.lang.String exception during replication

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.7, 7.7.1, 8.0
• Fix Version/s: 7.7.2, 8.1
• Component/s: replication (java), SolrCloud, SolrJ
• Labels: newbie replication

Environment

centos 7
solrcloud 7.7.1, 8.1.0

• Assignee: Noble Paul
• Reporter: Karl Stoney
• Created: 01/Mar/19 18:04
• Updated: 12/Jun/19 15:54
• Resolved: 22/Apr/19 08:41

Description

Since upgrading to 7.7 (also tried 7.7.1, and 8.1.0) from 6.6.4, we're seeing the following errors in the SolrCloud elected master for a given collection when updates are written. This was after a full reindex of data (fresh build).

request: http://solr-1.search-solr.preprod.k8.atcloud.io:80/solr/at-uk_shard1_replica_n2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsolr-2.search-solr.preprod.k8.atcloud.io%3A80%2Fsolr%2Fat-uk_shard1_replica_n1%2F&wt=javabin&version=2
Remote error message: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:385) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:183) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Following this through to the replica, you'll see:

08:35:22.060 [qtp1540374340-20] ERROR org.apache.solr.servlet.HttpSolrCall - null:java.lang.ClassCastException: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
    at org.apache.solr.common.util.JavaBinCodec.readEnumFieldValue(JavaBinCodec.java:813)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:339)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
    at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:640)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:337)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
    at org.apache.solr.common.util.JavaBinCodec.readMapEntry(JavaBinCodec.java:819)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:341)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:295)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
    at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
    at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
    at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
    at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
    at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.lang.Thread.run(Thread.java:748)

I noticed there some similar issues raised in 7x (https://issues.apache.org/jira/browse/SOLR-13255), so I don't know if this is the same sort of thing?

I've tried solr-users mailing list, and stackoverflow but to no avail. Please let me know if there is any other information I can provide.

Issue Links

• is blocked by: SOLR-13331 Atomic Update Multivalue remove does not work