Solr
  1. Solr
  2. SOLR-8422

Basic Authentication plugin is not working correctly in solrcloud

    Details

    • Flags:
      Important

      Description

      Iam seeing a problem with basic auth on Solr5.3.1 . We have 5 node solrcloud with basic auth configured on sgdsolar1/2/3/4/7 , listening on port 8984. We have 64 collections, each having 2 replicas distributed across the 5 servers in the solr cloud. A sample screen shot of the collections/shard locations shown below:-

      Step 1 - Our solr indexing tool sends a request to say any one of the solr servers in the solrcloud and the request is sent to a server which doesn't have the collection
      Here is the request sent by the indexing tool to sgdsolar1, that includes the correct BasicAuth credentials

      Step2 - Now sgdsolar1 routes the request to sgdsolar2 that has the collection1, but no basic auth header is being passed.

      As a results sgdsolar2 throws a 401 error back to source server sgdsolar1 and all the way back to solr indexing tool
      9.32.182.53 - - [15/Dec/2015:00:45:18 +0000] "GET /solr/collection1/get?route=Q049c2dkbWFpbDMwL089U0dfVVMx20093510!&ids=Q049c2dkbWFpbDMwL089U0dfVVMx20093510!08D9EACCA5AE663400257EB6005A5CFF,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!9057B828F841C41F00257EB6005A7421,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!F3FB9305A00A0E1200257EB6005AAA99,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!E9815A6F3CBC3D0E00257EB6005ACA02,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!FEB43AC9F648AFC500257EB6005AE4EB,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!4CF37E73A18F9D9F00257E590016CBD9,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!61D5457FEA1EBE5C00257E5900188729,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!6B0D89B9A7EEBC4600257E590019CEDA,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!360B9B52D9C6DFE400257EB2007FCD8B,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!D86D4CED01F66AF300257EB2008305A4&fl=unid,sequence,folderunid&wt=xml&rows=10 HTTP/1.1" 401 366

      2015-12-15 00:45:18.112 INFO (qtp1214753695-56) [c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@5ebe8fca
      2015-12-15 00:45:18.113 INFO (qtp1214753695-56) [c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [READ], collections: [collection1,], Path: [/get] path : /get params :fl=unid,sequence,folderunid&ids=Q049c2dkbWFpbDMwL089U0dfVVMx20093510!08D9EACCA5AE663400257EB6005A5CFF,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!9057B828F841C41F00257EB6005A7421,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!F3FB9305A00A0E1200257EB6005AAA99,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!E9815A6F3CBC3D0E00257EB6005ACA02,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!FEB43AC9F648AFC500257EB6005AE4EB,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!4CF37E73A18F9D9F00257E590016CBD9,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!61D5457FEA1EBE5C00257E5900188729,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!6B0D89B9A7EEBC4600257E590019CEDA,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!360B9B52D9C6DFE400257EB2007FCD8B,Q049c2dkbWFpbDMwL089U0dfVVMx20093510!D86D4CED01F66AF300257EB2008305A4&rows=10&wt=xml&route=Q049c2dkbWFpbDMwL089U0dfVVMx20093510!

      Step 3 - In another solrcloud , if the indexing tool sends the solr get request to the server that has the collection1, I see that basic authentication working as expected.

      I double checked and see both sgdsolar1/sgdsolar2 servers have the patched solr-core and solr-solrj jar files under the solr-webapp folder that were provided via earlier patches that Anshum/Noble worked on:-
      SOLR-8167 fixes the POST issue
      SOLR-8326 fixing PKIAuthenticationPlugin.
      SOLR-8355

        Activity

        Hide
        Ishan Chattopadhyaya added a comment -

        Nirmala, just to make sure we're not missing something, can you try the 5.4 release and see if this can be reproduced?
        Btw, did you mean to attach a screenshot with the ticket?

        Show
        Ishan Chattopadhyaya added a comment - Nirmala, just to make sure we're not missing something, can you try the 5.4 release and see if this can be reproduced? Btw, did you mean to attach a screenshot with the ticket?
        Hide
        Nirmala Venkatraman added a comment - - edited

        Noble - We tried your patch and it works with our indexing now. But occassionally while indexing I do see 401 Exceptions still.
        Here is solr.log snippet thats throwing the 401 error, as part of ConcurrentUpdateSolrClient

        2015-12-17 18:22:37.262 WARN (updateExecutor-2-thread-78-processing-http:////sgdsolar4.swg.usma.ibm.com:8983//solr//collection54_shard1_replica2 x:collection54_shard1_replica1 r:core_node1 n:sgdsolar1.swg.usma.ibm.com:8983_solr s:shard1 c:collection54) [c:collection54 s:shard1 r:core_node1 x:collection54_shard1_replica1] o.a.s.c.s.i.ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar4.swg.usma.ibm.com:8983/solr/collection54_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
        2015-12-17 18:22:37.264 ERROR (updateExecutor-2-thread-78-processing-http:////sgdsolar4.swg.usma.ibm.com:8983//solr//collection54_shard1_replica2 x:collection54_shard1_replica1 r:core_node1 n:sgdsolar1.swg.usma.ibm.com:8983_solr s:shard1 c:collection54) [c:collection54 s:shard1 r:core_node1 x:collection54_shard1_replica1] o.a.s.u.StreamingSolrClients error
        org.apache.solr.common.SolrException: Unauthorized request, Response code: 401

        request: http://sgdsolar4.swg.usma.ibm.com:8983/solr/collection54_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar1.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection54_shard1_replica1%2F&wt=javabin&version=2
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        Will this cause any real issues with index replicas?

        Show
        Nirmala Venkatraman added a comment - - edited Noble - We tried your patch and it works with our indexing now. But occassionally while indexing I do see 401 Exceptions still. Here is solr.log snippet thats throwing the 401 error, as part of ConcurrentUpdateSolrClient 2015-12-17 18:22:37.262 WARN (updateExecutor-2-thread-78-processing- http:////sgdsolar4.swg.usma.ibm.com:8983//solr//collection54_shard1_replica2 x:collection54_shard1_replica1 r:core_node1 n:sgdsolar1.swg.usma.ibm.com:8983_solr s:shard1 c:collection54) [c:collection54 s:shard1 r:core_node1 x:collection54_shard1_replica1] o.a.s.c.s.i.ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar4.swg.usma.ibm.com:8983/solr/collection54_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format 2015-12-17 18:22:37.264 ERROR (updateExecutor-2-thread-78-processing- http:////sgdsolar4.swg.usma.ibm.com:8983//solr//collection54_shard1_replica2 x:collection54_shard1_replica1 r:core_node1 n:sgdsolar1.swg.usma.ibm.com:8983_solr s:shard1 c:collection54) [c:collection54 s:shard1 r:core_node1 x:collection54_shard1_replica1] o.a.s.u.StreamingSolrClients error org.apache.solr.common.SolrException: Unauthorized request, Response code: 401 request: http://sgdsolar4.swg.usma.ibm.com:8983/solr/collection54_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar1.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection54_shard1_replica1%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Will this cause any real issues with index replicas?
        Hide
        Noble Paul added a comment -

        just to confirm, all nodes are updated with the patch , right?

        Show
        Noble Paul added a comment - just to confirm, all nodes are updated with the patch , right?
        Hide
        ASF subversion and git services added a comment -

        Commit 1721142 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1721142 ]

        SOLR-8422: When authentication enabled, requests fail if sent to a node that doesn't host
        the collection

        Show
        ASF subversion and git services added a comment - Commit 1721142 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1721142 ] SOLR-8422 : When authentication enabled, requests fail if sent to a node that doesn't host the collection
        Hide
        ASF subversion and git services added a comment -

        Commit 1721144 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1721144 ]

        SOLR-8422: When authentication enabled, requests fail if sent to a node that doesn't host
        the collection

        Show
        ASF subversion and git services added a comment - Commit 1721144 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1721144 ] SOLR-8422 : When authentication enabled, requests fail if sent to a node that doesn't host the collection
        Hide
        Anshum Gupta added a comment -

        If this is fixed, I'll backport this to be released with 5.3.2.

        Kindly let me know if that's the case or if we need to wait for this before releasing 5.3.2. I would rather wait and release with the fix.

        Nirmala Venkatraman and Noble Paul can you confirm?

        Show
        Anshum Gupta added a comment - If this is fixed, I'll backport this to be released with 5.3.2. Kindly let me know if that's the case or if we need to wait for this before releasing 5.3.2. I would rather wait and release with the fix. Nirmala Venkatraman and Noble Paul can you confirm?
        Hide
        Noble Paul added a comment -

        The original issue is fixed. We probably need to open another ticket for the on going TTL problem

        Show
        Noble Paul added a comment - The original issue is fixed. We probably need to open another ticket for the on going TTL problem
        Hide
        Anshum Gupta added a comment -

        Thanks for confirming. I'll backport this to lucene_solr_5_3.

        Show
        Anshum Gupta added a comment - Thanks for confirming. I'll backport this to lucene_solr_5_3.
        Hide
        ASF subversion and git services added a comment -

        Commit 1722120 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_3'
        [ https://svn.apache.org/r1722120 ]

        SOLR-8422: When authentication enabled, requests fail if sent to a node that doesn't host the collection (backport from branch_5x for 5.3.2 release)

        Show
        ASF subversion and git services added a comment - Commit 1722120 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1722120 ] SOLR-8422 : When authentication enabled, requests fail if sent to a node that doesn't host the collection (backport from branch_5x for 5.3.2 release)
        Hide
        Noble Paul added a comment -
        Show
        Noble Paul added a comment - I have opened SOLR-8470 Nirmala Venkatraman and Anshum Gupta
        Hide
        ASF subversion and git services added a comment -

        Commit 1722122 from Anshum Gupta in branch 'dev/trunk'
        [ https://svn.apache.org/r1722122 ]

        SOLR-8422: Add change log entry to 5.3.2 section on trunk

        Show
        ASF subversion and git services added a comment - Commit 1722122 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1722122 ] SOLR-8422 : Add change log entry to 5.3.2 section on trunk
        Hide
        ASF subversion and git services added a comment -

        Commit 1722124 from Anshum Gupta in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1722124 ]

        SOLR-8422: Add change log entry to 5.3.2 section (merge from trunk)

        Show
        ASF subversion and git services added a comment - Commit 1722124 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1722124 ] SOLR-8422 : Add change log entry to 5.3.2 section (merge from trunk)
        Hide
        Anshum Gupta added a comment -

        Marking this as resolved. Thanks everyone.

        Show
        Anshum Gupta added a comment - Marking this as resolved. Thanks everyone.
        Hide
        Nirmala Venkatraman added a comment -

        Paul - The patch for 8422 is applied on all 5 nodes and we are still seeing 401 exceptions. I haven't tried the solr-8470 patch (ttl property change) yet.

        Time (Local) Level Core Logger Message
        1/4/2016, 1:41:57 AM WARN null ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2,​ but 60) or the data in not in 'javabin' format
        1/4/2016, 1:41:57 AM ERROR null StreamingSolrClients error
        org.apache.solr.common.SolrException: Unauthorized request, Response code: 401

        request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        1/4/2016, 1:41:58 AM WARN null DistributedUpdateProcessor Error sending update to http://sgdsolar2.swg.usma.ibm.com:8983/solr
        1/4/2016, 1:41:58 AM WARN null ZkController Leader is publishing core=collection41_shard1_replica2 coreNodeName =core_node1 state=down on behalf of un-reachable replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/; forcePublishState? false
        1/4/2016, 1:41:58 AM ERROR null DistributedUpdateProcessor Setting up to try to start recovery on replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/col

        Show
        Nirmala Venkatraman added a comment - Paul - The patch for 8422 is applied on all 5 nodes and we are still seeing 401 exceptions. I haven't tried the solr-8470 patch (ttl property change) yet. Time (Local) Level Core Logger Message 1/4/2016, 1:41:57 AM WARN null ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2,​ but 60) or the data in not in 'javabin' format 1/4/2016, 1:41:57 AM ERROR null StreamingSolrClients error org.apache.solr.common.SolrException: Unauthorized request, Response code: 401 request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 1/4/2016, 1:41:58 AM WARN null DistributedUpdateProcessor Error sending update to http://sgdsolar2.swg.usma.ibm.com:8983/solr 1/4/2016, 1:41:58 AM WARN null ZkController Leader is publishing core=collection41_shard1_replica2 coreNodeName =core_node1 state=down on behalf of un-reachable replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/ ; forcePublishState? false 1/4/2016, 1:41:58 AM ERROR null DistributedUpdateProcessor Setting up to try to start recovery on replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/col
        Hide
        Noble Paul added a comment -

        Can you check the logs from the server who sent the 401 response?

        Show
        Noble Paul added a comment - Can you check the logs from the server who sent the 401 response?
        Hide
        Nirmala Venkatraman added a comment -

        sgdsolar7 has collection41_shard1_replica1
        sgdsolar2(9.32.182.60) has collection41_shard1_replica2
        In sgdsolar7's access logs, Iam seeing the last update from sgdsolar2 for collection41_shard_replica1 to server sgdsolar7 at 6:41:52 AM GMT. The above concurrentupdate error in solr.log on sgdsolar7 was thrown at 6:41:58 AM GMT

        9.32.182.60 - - [04/Jan/2016:06:41:40 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:41 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:42 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:42 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:46 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:46 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:50 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:50 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:52 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:41:52 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:42:06 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:42:06 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:42:08 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        9.32.182.60 - - [04/Jan/2016:06:42:08 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2
        Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 -
        -More-

        Show
        Nirmala Venkatraman added a comment - sgdsolar7 has collection41_shard1_replica1 sgdsolar2(9.32.182.60) has collection41_shard1_replica2 In sgdsolar7's access logs, Iam seeing the last update from sgdsolar2 for collection41_shard_replica1 to server sgdsolar7 at 6:41:52 AM GMT. The above concurrentupdate error in solr.log on sgdsolar7 was thrown at 6:41:58 AM GMT 9.32.182.60 - - [04/Jan/2016:06:41:40 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:41 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:42 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:42 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:46 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:46 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:50 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:50 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:51 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:52 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:41:52 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:42:06 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:42:06 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:42:08 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - 9.32.182.60 - - [04/Jan/2016:06:42:08 +0000] "POST /solr/collection41_shard1_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsgdsolar2.swg.usma.ibm.com%3A8983%2 Fsolr%2Fcollection41_shard1_replica2%2F&wt=javabin&version=2 HTTP/1.1" 200 - - More -
        Hide
        Anshum Gupta added a comment -

        Can you also post the stacktrace ?

        Show
        Anshum Gupta added a comment - Can you also post the stacktrace ?
        Hide
        Nirmala Venkatraman added a comment -

        Anshum - Here are the stack traces you asked in your earlier comment:-
        2016-01-04 06:38:25.531 WARN (qtp244229783-6448) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.e.j.h.HttpParser badMessage: java.lang.IllegalStateException: too much data after closed for HttpChannelOverHttp@40d64f82

        {r=117,c=false,a=IDLE,uri=-}

        2016-01-04 06:41:57.858 WARN (updateExecutor-2-thread-973-processing-http:////sgdsolar2.swg.usma.ibm.com:8983//solr//collection41_shard1_replica2 x:collection41_shard1_replica1 r:core_node2 n:sgdsolar7.swg.usma.ibm.com:8983_solr s:shard1 c:collection41) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.c.s.i.ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
        2016-01-04 06:41:57.883 ERROR (updateExecutor-2-thread-973-processing-http:////sgdsolar2.swg.usma.ibm.com:8983//solr//collection41_shard1_replica2 x:collection41_shard1_replica1 r:core_node2 n:sgdsolar7.swg.usma.ibm.com:8983_solr s:shard1 c:collection41) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.StreamingSolrClients error
        org.apache.solr.common.SolrException: Unauthorized request, Response code: 401

        request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        2016-01-04 06:41:58.194 WARN (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Error sending update to http://sgdsolar2.swg.usma.ibm.com:8983/solr
        org.apache.solr.common.SolrException: Unauthorized request, Response code: 401

        request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        2016-01-04 06:41:58.204 WARN (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.c.ZkController Leader is publishing core=collection41_shard1_replica2 coreNodeName =core_node1 state=down on behalf of un-reachable replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/; forcePublishState? false
        2016-01-04 06:41:58.207 ERROR (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/
        org.apache.solr.common.SolrException: Unauthorized request, Response code: 401

        request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        Show
        Nirmala Venkatraman added a comment - Anshum - Here are the stack traces you asked in your earlier comment:- 2016-01-04 06:38:25.531 WARN (qtp244229783-6448) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.e.j.h.HttpParser badMessage: java.lang.IllegalStateException: too much data after closed for HttpChannelOverHttp@40d64f82 {r=117,c=false,a=IDLE,uri=-} 2016-01-04 06:41:57.858 WARN (updateExecutor-2-thread-973-processing- http:////sgdsolar2.swg.usma.ibm.com:8983//solr//collection41_shard1_replica2 x:collection41_shard1_replica1 r:core_node2 n:sgdsolar7.swg.usma.ibm.com:8983_solr s:shard1 c:collection41) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.c.s.i.ConcurrentUpdateSolrClient Failed to parse error response from http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format 2016-01-04 06:41:57.883 ERROR (updateExecutor-2-thread-973-processing- http:////sgdsolar2.swg.usma.ibm.com:8983//solr//collection41_shard1_replica2 x:collection41_shard1_replica1 r:core_node2 n:sgdsolar7.swg.usma.ibm.com:8983_solr s:shard1 c:collection41) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.StreamingSolrClients error org.apache.solr.common.SolrException: Unauthorized request, Response code: 401 request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-01-04 06:41:58.194 WARN (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Error sending update to http://sgdsolar2.swg.usma.ibm.com:8983/solr org.apache.solr.common.SolrException: Unauthorized request, Response code: 401 request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-01-04 06:41:58.204 WARN (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.c.ZkController Leader is publishing core=collection41_shard1_replica2 coreNodeName =core_node1 state=down on behalf of un-reachable replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/ ; forcePublishState? false 2016-01-04 06:41:58.207 ERROR (qtp244229783-6619) [c:collection41 s:shard1 r:core_node2 x:collection41_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/ org.apache.solr.common.SolrException: Unauthorized request, Response code: 401 request: http://sgdsolar2.swg.usma.ibm.com:8983/solr/collection41_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsgdsolar7.swg.usma.ibm.com%3A8983%2Fsolr%2Fcollection41_shard1_replica1%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
        Hide
        Nirmala Venkatraman added a comment -

        I applied Noble's patch for pkiauth.ttl(SOLR-8470) and set the ttl parameter to 60sec(default is 5sec) and ran another batch of indexing load. Good news is that I didn't hit any of the 401 exceptions , but one of the nodes sgdsolar7 went into recovery with zksession expiration in /overseer/elect.

        Show
        Nirmala Venkatraman added a comment - I applied Noble's patch for pkiauth.ttl( SOLR-8470 ) and set the ttl parameter to 60sec(default is 5sec) and ran another batch of indexing load. Good news is that I didn't hit any of the 401 exceptions , but one of the nodes sgdsolar7 went into recovery with zksession expiration in /overseer/elect.
        Hide
        ASF subversion and git services added a comment -

        Commit 1724193 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4'
        [ https://svn.apache.org/r1724193 ]

        SOLR-8422: When authentication enabled, requests fail if sent to a node that doesn't host the collection.

        Show
        ASF subversion and git services added a comment - Commit 1724193 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4' [ https://svn.apache.org/r1724193 ] SOLR-8422 : When authentication enabled, requests fail if sent to a node that doesn't host the collection.
        Hide
        ASF subversion and git services added a comment -

        Commit 9ef144ddefe21f30c1c9ebd5246e7e03387488e1 in lucene-solr's branch refs/heads/branch_5_4 from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ef144d ]

        SOLR-8460, SOLR-8373, SOLR-8422, SOLR-7462, SOLR-8470: Add CHANGES entries for 5.4.1.

        git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_4@1724198 13f79535-47bb-0310-9956-ffa450edef68

        Show
        ASF subversion and git services added a comment - Commit 9ef144ddefe21f30c1c9ebd5246e7e03387488e1 in lucene-solr's branch refs/heads/branch_5_4 from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ef144d ] SOLR-8460 , SOLR-8373 , SOLR-8422 , SOLR-7462 , SOLR-8470 : Add CHANGES entries for 5.4.1. git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_4@1724198 13f79535-47bb-0310-9956-ffa450edef68
        Hide
        Varun Thacker added a comment -

        Adding a BasicAuth tag

        Show
        Varun Thacker added a comment - Adding a BasicAuth tag

          People

          • Assignee:
            Noble Paul
            Reporter:
            Nirmala Venkatraman
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development