[SOLR-15317] Parts of internal SolR communication fail when the CertAuthPlugin is active

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.8.1
• Fix Version/s: 8.9, 8.8.3
• Component/s: Authentication

Environment

OS: CentOS 7

Java: JDK 11

• Assignee: Mike Drob
• Reporter: Dominik Dresel
• Created: 06/Apr/21 06:33
• Updated: 22/Mar/22 20:19
• Resolved: 09/Apr/21 15:21

Description

Hi all!

While I was testing out the CertAuthPlugin for the new SolR 9 it came to my attention that various internal HTTP calls in SolR fail. For example when I try to add a BinaryResponseWriter via curl it fails with lots of authentication errors (HTTP status code 401). Other actions (like creating schema fields for collections) via curl work fine. To reproduce the problem, following steps have to be taken (on Linux):

• git clone https://github.com/apache/solr.git (I used commit caf8cbc0aa11e32f894a90531e3e9f20edf75efa)
• cd solr
• ./gradlew assemble
• cd solr/packaging/build/solr-9.0.0-SNAPSHOT/
• keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:127.0.0.1 -dname "CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country"
• openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.key -nodes -nocerts
• openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.crt -nodes -nokeys
• echo 'SOLR_SSL_ENABLED=true' >> bin/solr.in.sh
• echo 'SOLR_SSL_KEY_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh
• echo 'SOLR_SSL_KEY_STORE_PASSWORD=secret' >> bin/solr.in.sh
• echo 'SOLR_SSL_TRUST_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh
• echo 'SOLR_SSL_TRUST_STORE_PASSWORD=secret' >> bin/solr.in.sh
• echo 'SOLR_SSL_NEED_CLIENT_AUTH=true' >> bin/solr.in.sh
• echo 'SOLR_SSL_WANT_CLIENT_AUTH=false' >> bin/solr.in.sh
• echo 'SOLR_SSL_CHECK_PEER_NAME=false' >> bin/solr.in.sh
• ./bin/solr start -v -c
• server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd clusterprop -name urlScheme -val https
• ./bin/solr zk cp file:///tmp/security.json zk:/security.json -z localhost:9983
• ./bin/solr stop
• ./bin/solr start -v -c
• ./bin/solr create -c testcollection
• curl --cacert ./solr-ssl.keystore.crt --key ./solr-ssl.keystore.key --cert ./solr-ssl.keystore.crt "https://localhost:8983/api/collections/testcollection/config" -H "Content-Type: application/json" --data-binary '{ "add-queryresponsewriter":{ "class":"solr.BinaryResponseWriter", "name":"test" }}'

 

After the last curl command (which takes about 30 seconds) the following error message is printed:

{{

{ "responseHeader":

{ "status":500, "QTime":30017}

, "errorMessages":["1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/\n"], "WARNING":"This response format is experimental. It is likely to change in the future.", "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/", "trace":"org.apache.solr.common.SolrException: 1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:829)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:549)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:381)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:140 )\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)\n\tat org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:269)\n\tat org.apache.solr.api.V2HttpCall.execute(V2HttpCall.java:354)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat org.eclipse.jetty.servlet. ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)\n\tat org.eclipse.j etty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:134 9)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.hand ler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.jav a:105)\n\tat org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:146)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n", "code":500}}}}

 

In the SolR WEB-UI lots of PKIAuthentication errors are printed (see solr_error.png). Out of curiosity I backported the CertAuthPlugin to SolR v8.8.1 locally and SolR 8 had the same issues as the current master.

Issue Links

• links to: GitHub Pull Request #64