Details
Blocker Description
Here are the steps that recreate this issue. I tried this on Solr 5.4 and I had the following stack trace when I issued an ADDREPLICA. This seems pretty similar to what we saw on SOLR-8326 so it might be just something we missed but we should make sure that we ship 5.4 with this fixed.
#Upload Security Conf
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd putfile /security.json ~/security.json
#Start Solr
bin/solr start -e cloud -z localhost:2181
#Collection Admin Edit Command:
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"collection-admin-edit", "role":"admin"}}'
#Read User and permission:
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read", "role":"read"}}'
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"update", "role":"update"]}}'
#Add Users
#Read User
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d '{"set-user" : {"solrread":"solrRocks"}}'
#Update user
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d '{"set-user" : {"solrupdate":"solrRocks"}}'
#Set user roles
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-user-role" : {"solrupdate":["read","update"]}}'
#Read User
curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-user-role" : {"solrread":["read"]}}'
#Create collection
curl --user solr:SolrRocks 'http://localhost:8983/solr/admin/collections?action=CREATE&name=a&numShards=1&replicationFactor=1&collection.configName=gettingstarted&wt=json'
#Add Replica
curl --user solr:SolrRocks 'http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=a&shard=shard1&wt=json'
Exception log:
INFO - 2015-12-01 04:57:47.022; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery.
INFO - 2015-12-01 04:57:47.023; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://172.20.10.4:7574/solr/a_shard1_replica1/.
ERROR - 2015-12-01 04:57:47.027; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.common.SolrException; Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.20.10.4:7574/solr/a_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Unauthorized request, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/a_shard1_replica1/update. Reason:
<pre> Unauthorized request, Response code: 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:205)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:145)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:436)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)
INFO - 2015-12-01 04:57:47.028; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping buffered updates FSUpdateLog
{state=BUFFERING, tlog=null}ERROR - 2015-12-01 04:57:47.028; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (4)