Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9188

BlockUnknown property makes inter-node communication impossible

    Details

      Description

      When I setup my solr cloud without blockUnknown property it works as expected. When I want to block non authenticated requests I get following stacktrace during startup (see attached file).

      1. SOLR-9188.patch
        3 kB
        Noble Paul
      2. solr9188-errorlog.txt
        34 kB
        Jan Høydahl

        Issue Links

          Activity

          Hide
          algalg@gmail.com Ales Gregor added a comment -

          I have the same issue. Is there any workaround that would allow me to secure everything in Solr with BasicAuth and avoided this error?

          Show
          algalg@gmail.com Ales Gregor added a comment - I have the same issue. Is there any workaround that would allow me to secure everything in Solr with BasicAuth and avoided this error?
          Hide
          alexandre.drouin Alex D added a comment -

          On my development workstation (windows) I was able to workaround the issue by editing solr.in.cmd and changing SOLR_HOST to include a username & password. e.g.:
          set SOLR_HOST=username:password@mysolrhost.com

          Show
          alexandre.drouin Alex D added a comment - On my development workstation (windows) I was able to workaround the issue by editing solr.in.cmd and changing SOLR_HOST to include a username & password. e.g.: set SOLR_HOST=username:password@mysolrhost.com
          Hide
          janhoy Jan Høydahl added a comment -

          Trying to dig deeper:

          Susheel Kumar also reported on the mailing list but says that his cluster works well, except from the error logs. Are you sure there are no side effects? In SOLR-9257, Martin Löper says "This works well when there is no inter-node communication. As soon as I create a collection with 2 shards, I get the following exception for every access of the "/select" request handler. .... Error 401 Unauthorized request..."

          In SOLR-9257, Shankar R says that removing blockUnknown does not help. Can you confirm this?

          I rased the priority of this to Critical, please help shed some more light on this

          Show
          janhoy Jan Høydahl added a comment - Trying to dig deeper: Susheel Kumar also reported on the mailing list but says that his cluster works well, except from the error logs. Are you sure there are no side effects? In SOLR-9257 , Martin Löper says "This works well when there is no inter-node communication. As soon as I create a collection with 2 shards, I get the following exception for every access of the "/select" request handler. .... Error 401 Unauthorized request..." In SOLR-9257 , Shankar R says that removing blockUnknown does not help. Can you confirm this? I rased the priority of this to Critical, please help shed some more light on this
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Yes, Jan. The The cluster in my case works fine without any issue and infact we moved up to three environment (development, functional & performance) with QA certified and didn't notice any issue until one developer noticed these error messages in Logs.

          Removing blockUnknown doesn't help as it then allows anyone to access Solr directly without challenging with user / pwd.

          The Solr Cluster in our case has multiple shards.

          Please let me know if i can provide any more details.

          Thanks,
          Susheel

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Yes, Jan. The The cluster in my case works fine without any issue and infact we moved up to three environment (development, functional & performance) with QA certified and didn't notice any issue until one developer noticed these error messages in Logs. Removing blockUnknown doesn't help as it then allows anyone to access Solr directly without challenging with user / pwd. The Solr Cluster in our case has multiple shards. Please let me know if i can provide any more details. Thanks, Susheel
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 44c30f0535ceed5f2ad08aa8a9f974d4973774e0 in lucene-solr's branch refs/heads/master from Noble Paul
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=44c30f0 ]

          SOLR-9188: blockUnknown property makes inter-node communication impossible

          Show
          jira-bot ASF subversion and git services added a comment - Commit 44c30f0535ceed5f2ad08aa8a9f974d4973774e0 in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=44c30f0 ] SOLR-9188 : blockUnknown property makes inter-node communication impossible
          Hide
          noble.paul Noble Paul added a comment -

          The problem is that 'blockUnknown=true' even blocks access to /admin/info/key which should be accessible freely w/o authentication

          Show
          noble.paul Noble Paul added a comment - The problem is that 'blockUnknown=true' even blocks access to /admin/info/key which should be accessible freely w/o authentication
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0ed8c2a7ad7038f99bff3322b06edf948a61dfe0 in lucene-solr's branch refs/heads/master from Noble Paul
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ed8c2a ]

          SOLR-9188: Trying revert a change and fix the unexpected IOException in jenkins failure.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0ed8c2a7ad7038f99bff3322b06edf948a61dfe0 in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ed8c2a ] SOLR-9188 : Trying revert a change and fix the unexpected IOException in jenkins failure.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 757c245bee057b899107be113fcfc0e4cce3b4a2 in lucene-solr's branch refs/heads/master from Noble Paul
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=757c245 ]

          SOLR-9188: Trying revert a change and fix the unexpected IOException in jenkins failure.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 757c245bee057b899107be113fcfc0e4cce3b4a2 in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=757c245 ] SOLR-9188 : Trying revert a change and fix the unexpected IOException in jenkins failure.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b3526c568ca03b7eb2d043a21373689413fa2e7f in lucene-solr's branch refs/heads/branch_6x from Noble Paul
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b3526c5 ]

          SOLR-9188: blockUnknown property makes inter-node communication impossible

          Show
          jira-bot ASF subversion and git services added a comment - Commit b3526c568ca03b7eb2d043a21373689413fa2e7f in lucene-solr's branch refs/heads/branch_6x from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b3526c5 ] SOLR-9188 : blockUnknown property makes inter-node communication impossible
          Hide
          noble.paul Noble Paul added a comment -

          Barking up the wrong tree. The tests were failing in master even before the fix

          Show
          noble.paul Noble Paul added a comment - Barking up the wrong tree. The tests were failing in master even before the fix
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Re-opened to back-port to 6.2.1

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Re-opened to back-port to 6.2.1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 988c2149802285a9be9f8036bf803ca610e27cad in lucene-solr's branch refs/heads/branch_6_2 from Noble Paul
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=988c214 ]

          SOLR-9188: blockUnknown property makes inter-node communication impossible

          (cherry picked from commit b3526c5)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 988c2149802285a9be9f8036bf803ca610e27cad in lucene-solr's branch refs/heads/branch_6_2 from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=988c214 ] SOLR-9188 : blockUnknown property makes inter-node communication impossible (cherry picked from commit b3526c5)
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Closing after 6.2.1 release

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.2.1 release
          Hide
          ewencluley Ewen Cluley added a comment -

          I have deployed 6.2.1 and am still encountering the same (i think the same) issue. I am using self signed ssl certificates but dont think that should make an impact.

          The work around still works where i specify adminuser:Password@servername.com as the solr host name in the solr.in.sh file.

          Log:
          2016-10-25 10:46:34.243 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://server00314.phx.abc.com:8984/solr
          org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-'
          at org.noggit.JSONParser.err(JSONParser.java:356)
          at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712)
          at org.noggit.JSONParser.next(JSONParser.java:886)
          at org.noggit.JSONParser.nextEvent(JSONParser.java:930)
          at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44)
          at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37)
          at org.apache.solr.common.util.Utils.fromJSON(Utils.java:108)
          at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:203)
          at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:156)
          at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118)
          at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
          at org.eclipse.jetty.server.Server.handle(Server.java:518)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          at java.lang.Thread.run(Thread.java:745)
          2016-10-25 10:46:34.243 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong
          java.security.InvalidKeyException: No installed provider supports this key: (null)
          at javax.crypto.Cipher.chooseProvider(Cipher.java:893)
          at javax.crypto.Cipher.init(Cipher.java:1249)
          at javax.crypto.Cipher.init(Cipher.java:1186)
          at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277)
          at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:173)
          at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:160)
          at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118)
          at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
          at org.eclipse.jetty.server.Server.handle(Server.java:518)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          at java.lang.Thread.run(Thread.java:745)
          2016-10-25 10:46:34.243 WARN (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Failed to decrypt header, trying after refreshing the key
          2016-10-25 10:46:34.245 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://server00314.phx.abc.com:8984/solr
          org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-'
          at org.noggit.JSONParser.err(JSONParser.java:356)
          at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712)
          at org.noggit.JSONParser.next(JSONParser.java:886)
          at org.noggit.JSONParser.nextEvent(JSONParser.java:930)
          at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44)
          at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37)
          at org.apache.solr.common.util.Utils.fromJSON(Utils.java:108)
          at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:203)
          at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:163)
          at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118)
          at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
          at org.eclipse.jetty.server.Server.handle(Server.java:518)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          at java.lang.Thread.run(Thread.java:745)
          2016-10-25 10:46:34.245 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong
          java.security.InvalidKeyException: No installed provider supports this key: (null)
          at javax.crypto.Cipher.chooseProvider(Cipher.java:893)
          at javax.crypto.Cipher.init(Cipher.java:1249)
          at javax.crypto.Cipher.init(Cipher.java:1186)
          at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277)
          at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:173)
          at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:164)
          at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118)
          at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
          at org.eclipse.jetty.server.Server.handle(Server.java:518)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          at java.lang.Thread.run(Thread.java:745)
          2016-10-25 10:46:34.246 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Could not decipher a header server00314.phx.abc.com:8984_solr HUT+T67HEIJCECx+v+yJ9eEhMfW8jGCW3n1kpnpPqC+iELA7mvKMskrbgtscJR7psQHzAU83SYopyB6ERsG8WQ== . No principal set

          Show
          ewencluley Ewen Cluley added a comment - I have deployed 6.2.1 and am still encountering the same (i think the same) issue. I am using self signed ssl certificates but dont think that should make an impact. The work around still works where i specify adminuser:Password@servername.com as the solr host name in the solr.in.sh file. Log: 2016-10-25 10:46:34.243 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://server00314.phx.abc.com:8984/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:108) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:203) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:156) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 2016-10-25 10:46:34.243 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.chooseProvider(Cipher.java:893) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:173) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:160) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 2016-10-25 10:46:34.243 WARN (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Failed to decrypt header, trying after refreshing the key 2016-10-25 10:46:34.245 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://server00314.phx.abc.com:8984/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:108) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:203) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:163) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 2016-10-25 10:46:34.245 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.chooseProvider(Cipher.java:893) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:173) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:164) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:313) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 2016-10-25 10:46:34.246 ERROR (qtp240650537-21) [c:ecm s:shard3 r:core_node2 x:ecm_shard3_replica1] o.a.s.s.PKIAuthenticationPlugin Could not decipher a header server00314.phx.abc.com:8984_solr HUT+T67HEIJCECx+v+yJ9eEhMfW8jGCW3n1kpnpPqC+iELA7mvKMskrbgtscJR7psQHzAU83SYopyB6ERsG8WQ== . No principal set
          Hide
          janhoy Jan Høydahl added a comment -

          This could perhaps be the bug I discovered in SOLR-9640?

          • Fix bug in SolrDispatchFilter - path /admin/info/key should always be open. It required authentication since we were comparing with getPathInfo instead of getServletPath
          Show
          janhoy Jan Høydahl added a comment - This could perhaps be the bug I discovered in SOLR-9640 ? Fix bug in SolrDispatchFilter - path /admin/info/key should always be open. It required authentication since we were comparing with getPathInfo instead of getServletPath
          Hide
          noble.paul Noble Paul added a comment -

          I figured it out. In our JUnit tests only getPathInfo() works and in normal webapp only getServletPath() works. So, the fix is to do both checks

          Show
          noble.paul Noble Paul added a comment - I figured it out. In our JUnit tests only getPathInfo() works and in normal webapp only getServletPath() works. So, the fix is to do both checks
          Hide
          noble.paul Noble Paul added a comment -

          its still broken

          Show
          noble.paul Noble Paul added a comment - its still broken
          Hide
          janhoy Jan Høydahl added a comment -

          Perhaps open a new JIRA to fix this since this one is already released in 6.2.1.

          Show
          janhoy Jan Høydahl added a comment - Perhaps open a new JIRA to fix this since this one is already released in 6.2.1.
          Hide
          ewencluley Ewen Cluley added a comment -

          Cool, thanks for the quick response too. I am unfamiliar with the workflow and release schedule used on the Solr project, am I correct in thinking this fix will be released as part of 6.3? If so, any idea of the release schedule and when this will be released? Anything I can do to help with testing? Thanks

          Show
          ewencluley Ewen Cluley added a comment - Cool, thanks for the quick response too. I am unfamiliar with the workflow and release schedule used on the Solr project, am I correct in thinking this fix will be released as part of 6.3? If so, any idea of the release schedule and when this will be released? Anything I can do to help with testing? Thanks
          Hide
          noble.paul Noble Paul added a comment - - edited

          Yes, it'll be there in the upcoming release (6.3) . The release should happen over the next 2-3 weeks

          Show
          noble.paul Noble Paul added a comment - - edited Yes, it'll be there in the upcoming release (6.3) . The release should happen over the next 2-3 weeks
          Hide
          ewencluley Ewen Cluley added a comment -

          Awesome. Will deploy the patch on top of my 6.2.1 and test that it resolved the issue for me. Thanks

          Show
          ewencluley Ewen Cluley added a comment - Awesome. Will deploy the patch on top of my 6.2.1 and test that it resolved the issue for me. Thanks
          Hide
          janhoy Jan Høydahl added a comment -

          Can we please leave this JIRA as fixed in 6.2.1 and close this? Then do the fix of regression bugs in SOLR-9692 in 6.3? This CHANGES history rewrite is just confusing in my eyes.

          Show
          janhoy Jan Høydahl added a comment - Can we please leave this JIRA as fixed in 6.2.1 and close this? Then do the fix of regression bugs in SOLR-9692 in 6.3? This CHANGES history rewrite is just confusing in my eyes.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment - - edited

          Can we please leave this JIRA as fixed in 6.2.1 and close this? Then do the fix of regression bugs in SOLR-9692 in 6.3? This CHANGES history rewrite is just confusing in my eyes.

          +1. Issues once resolved and closed (released) should not be opened and new jira should be used for further work. Please return this issue to its former state i.e. fixed as of 6.2.1. Please do not rewrite CHANGES.txt as it is not a living document. It is part of every release artifact as well as archived on the site e.g. https://lucene.apache.org/solr/6_2_1/changes/Changes.html. If you want, you can mention this issue as part of the CHANGES.txt in 6.3 saying that SOLR-9188 did not fix the issue as stated in 6.2.1

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - - edited Can we please leave this JIRA as fixed in 6.2.1 and close this? Then do the fix of regression bugs in SOLR-9692 in 6.3? This CHANGES history rewrite is just confusing in my eyes. +1. Issues once resolved and closed (released) should not be opened and new jira should be used for further work. Please return this issue to its former state i.e. fixed as of 6.2.1. Please do not rewrite CHANGES.txt as it is not a living document. It is part of every release artifact as well as archived on the site e.g. https://lucene.apache.org/solr/6_2_1/changes/Changes.html . If you want, you can mention this issue as part of the CHANGES.txt in 6.3 saying that SOLR-9188 did not fix the issue as stated in 6.2.1
          Hide
          noble.paul Noble Paul added a comment - - edited

          I changed that CHANGES.txt rewrite

          Show
          noble.paul Noble Paul added a comment - - edited I changed that CHANGES.txt rewrite
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Closing after 6.3.0 release.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

            People

            • Assignee:
              noble.paul Noble Paul
              Reporter:
              piotr.tempes@igt.com Piotr Tempes
            • Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development